Nelson Silva is a co-founder of inEvo and an active member of the Nuxeo community. He answers questions on answers.nuxeo.com, has published some sample Nuxeo code on GitHub (like a Dart Gadget), and even participated in our recent Bug Day. He contributed code for Tomcat 7 support and is working on OAuth2 and a Shindig upgrade.

You are one of the founders of a Portuguese company called inEvo, which specializes in web applications, computer graphics, business intelligence and software integration. Can you tell us more about the company?

Tiago Cardoso and I were both working as researchers in INESC-ID Lisbon (a research center here in Portugal) where we had the opportunity to participate in several European research projects. We decided to do it for a living so we founded inEvo, a research oriented company. Along the way we connected with some local IT companies which contracted us to do consulting and integration work so we gained experience in a wide range of applications; this allowed us to fund our in-house research.

One of our first results was a Flash-based SVG library, getting us our first international clients with whom we developed a wide range of applications, from online postcard customizations to aircraft parking planning. Our goal is to innovate and provide state-of-the-art solutions to our clients. We have a strong focus on open source and strive to give back to the community whenever possible.

You’ve been working on the Nuxeo Platform for three years now and have built an application called ColaDI. Can you tell us more about this project? Why did you choose the Nuxeo Platform to help build this application?

ColaDI (http://coladi.inevo.pt/) was our first research project partially funded by the local innovation agency. It is a collaborative document management solution for industrial design, focusing on three main areas:
- 2D and 3D document management
- Content-based retrieval for 2D and 3D content
- Real-time collaboration

Nuxeo was an obvious choice for document management. The platform is open source, modular, extensible and has proper feature “blocks” that we can assemble as needed. Nuxeo allowed us to easily create custom types for the 2D and 3D content. It also supports CMIS, which was adopted to allow for easy integration with other ECM solutions.

The 2D and 3D content retrieval modules were built on our own custom engine called EnContRA (an Engine for Content Retrieval Applications) which was developed throughout the project. We index the content stored in Nuxeo and use a custom page provider that delegates to EnContra’s search engine. We support querying 2D and 3D content by sketch and by example.

For real-time collaboration, we used Apache Wave and created a module that allows users to collaboratively discuss and annotate 2D and 3D content. We also have a generator for meeting notes based on simple annotations within the discussion.

It looks like you have a CAD viewer for 2D and 3D, with SVG-based annotations. How did you extend Nuxeo to achieve this?

The first thing we did was select a target format for both 2D and 3D content to work with. We needed a declarative format that could easily be manipulated in the browser so we chose SVG for 2D and X3D for 3D — both XML-based.
We then plugged our conversion pipeline to Nuxeo by contributing new converters. For 2D content, we added PSD and DXF support and for 3D, we added IGES, BREP and STEP support by relying on OpenCascade and VTK. To view the content, we contributed a couple of preview adapters so we have both image and SVG previews for 2D documents as well as an image-based swivel and X3D preview for 3D. We’re relying on browser support for SVG, and for X3D, we’re using X3DOM which uses WebGL with a Flash-based fallback.

The annotation application is similar to Nuxeo’s and is also built with GWT. We used lib-gwt-svg to handle SVG and developed lib-gwt-x3d to work with X3D. We also added a gesture recognizer which supports commands (like delete, select, etc.) and some predefined shapes (circle, rectangle, triangle, etc.).

All the annotations use SVG and are stored in Nuxeo’s Annotea server. We took Nuxeo’s example: We added new XPointer functions to support our 3D annotations so we have functions to add camera-based annotations where you can draw over the whole scene, and object annotations, where you can anchor the annotation to a 3D point and draw in a billboard.

Since both the annotations and their content use XML (RDF and SVG) we were able to use them as the model for the collaborative annotation server which relies on Apache Wave.

You’re currently working on an upgrade of Nuxeo’s OpenSocial container, and on an OAuth2 plugin. That’s awesome! Can you tell us about this?

Basically, I think Nuxeo should at least provide a basic OAuth2 consumer framework. This will enable some cool integrations. Most of the existing providers are using OAuth2 so I started working on a very simple consumer framework based on the existing OAuth support and using Google’s OAuth2 library.

The upgrade of Nuxeo’s OpenSocial container was more of a requirement since it dates back to 2009 and relies on some deprecated libraries which conflict with newer versions needed by Google’s OAuth2 library. I branched Nuxeo’s OpenSocial module and upgraded to the latest Shindig version but I’m not, by any means, an expert on Shindig, so this will probably need some work by the original contributors of the OpenSocial module and by Nuxeo’s experts before being merged. I can also patch the current used Shindig version to remove the deprecated libraries, which should require fewer changes to Nuxeo, but I’ll leave it up to you guys to decide.

I’ve also started to work on an OpenID Connect module which basically supports authentication through OAuth2. I’ve added two providers as an example (Google and Facebook) so now I can log in to Nuxeo using my Google or Facebook credentials.

Rumor has it that you have another project based on the Nuxeo Platform starting soon. Can you tell us more about this?

inEvo was recently selected to develop a social platform for a research center here in Portugal. We proposed to develop all the modules using Nuxeo instead of relying of several different projects. Consequently, we’ll need to come up with simple WCM features (pages, posts, etc.), a collaborative Wiki, some LMS features, conferencing/broadcasting support, a Q&A system, 3D and video content support, etc.

We’ll be relying on WebEngine to develop the public facing features and we’ll integrate several third party solutions with Nuxeo along the way (Etherpad Lite, BigBlueButton/Google Hangout, YouTube, etc..) so we’ll probably have something to give back once this is completed.

I saw some pretty cool projects in the inEvo Labs. What is inEvo Labs? Are you planning to open source those projects?

Like I said before: We have a strong focus on research, and if we had resources that’d be all we’d do. We are always experimenting with new ideas and technologies and felt we should showcase some of our research results so we created inEvo Labs.

We are strong supporters of open source and would love to open source most of these projects. The sad truth is that we haven’t found time to do this. As you’re aware, open sourcing a project requires a commitment to the community and right now, we cannot afford it. In the near future we’ll probably release these and other projects as open source, so it really is just a matter of time. In the meantime, feel free to take a look at our GitHub, since we have several other projects there which already keep us very busy.

Speaking of open source, I saw on your blog that you are vert.x users. What do you think about how VMware handled Tim Fox’s departure and the vert.x project?

vert.x was a project that caught my attention as soon as it was released. I love its polyglot nature and the concept of verticals with a message bus which provide a modular extensible framework.

I contributed to vert.x with the CommonJS support and created node.vert.x which is a simple node.js compatibility layer. As an experiment I got ShareJS working on top of Vert.x by creating connect.vert.x and sockjs-node.vert.x.

Getting back to the original question, I think this situation escalated further than it should. Tim Fox was an employee at VMware so it makes sense that ultimately they are the owners of vert.x. The problem here was the tone of the letter Tim Fox got from VMware’s lawyers, but I guess these are common legal procedures and should not be seen as VMware’s position on the subject. In the end, everything seems to have come together and vert.x will become an Eclipse project.

As a side note: I’m really looking forward to seeing what Nashorn and it’s Node.js support bring to the project…

If you could have three wishes for the Nuxeo Platform, what would they be?

I guess most of my wishes are already in the roadmap: upgrade to JEE6, support for Tomcat 7 and use other view technologies.
If I had to come up with some new ones, I’d say:
- Provide OAuth2 and OpenID Connect support (shameless plug).
- Adopt Dart and web components and replace GWT. The Dart IDE is based on Eclipse, which would also guarantee proper integration with Nuxeo IDE.
- Make hot fixes freely available. Not all customers need Nuxeo Studio, but regular updates are essential for wider adoption.

Nuxeo: Hot fixes are freely available – the source code is open, the recipe to build them is documented, and the binaries are freely downloadable. Hot fix packaging is actually an added-value service, and part of the Nuxeo Connect subscription. Nuxeo’s business model is to derive revenue from maintenance, support and added-value services (such as Nuxeo Studio) provided within a service subscription.

What do you like to do when you’re not working?

When you work for yourself it’s often difficult to maintain a clear working/non-working schedule. But, with a 3-year-old and a 2-month-old, I guess what I like to do the most is just play with them.

I also like to play electric guitar and read, although most of my bedtime reading includes technical books, which could be considered “working.”

The post [Nuxeo Community] Meet Nelson Silva appeared first on Nuxeo Blog.

if it’s possible to deploy a contribution on per method base

Today we have a question from Edgar who asks if it’s possible to deploy a contribution on per method basis? I like this question because the use of the Deploy annotation can be misleading. Here’s why: You can put this annotation on a method and it won’t do anything, which is deeply disturbing to me. That’s because method annotations are never looked for in our JUnit runner (One day I’ll have to fix the target of those annotations).

So if you cannot use the Deploy and LocalDeploy annotations on your method, the question is: How do you deploy bundles or contributions inside them? Well, the proper way to do this is to use the test harness you can inject in your test using the @Inject annotation.

Injection is one of the most important part of our test framework. But, it’s not easy to know what you can/cannot inject. To understand this better, let’s talk about Guice. It’s the library used for injection. Know that you can only inject what you have manually bound to the injector. So if you can inject the test harness, it means someone has already bound it for you. This has happened in the RuntimeFeature.configure method (itself called by the FeatureRunner). In this method, every service deployed (which you can retrieve like this: Framework.getRuntime().getComponentManager().getServices()) are bound as well as the test harness. This is all you’ll be able to inject during the test execution. To know what you can inject, look for the configure method of your test features.

Back to our first question, how to deploy bundles or contributions on a per method basis. Here’s how:

The post [Q&A Friday] How to deploy a contribution on a per method basis? appeared first on Nuxeo Blog.

The Nuxeo Platform hotfix 5.6.0-HF10 is out and available for Nuxeo Connect clients from the Admin Center or Nuxeo Marketplace. It’s also available in the maintenance branch of the source code tree.

Bug

• [NXP-10801] – </HTML> tag added systematically on a rendition generated using freemarker engine
• [NXP-10873] – Dynamically generate graph container id from the widget id
• [NXP-10877] – Text download (green arrow) on Note documents truncates content with accentuated chars
• [NXP-10889] – Tomcat AJP Connector configuration omits emptySessionPath attribute
• [NXP-10892] – Duplicated javassist lib in test classpath
• [NXP-10929] – Remove permission should allow to put a document to the trash
• [NXP-10933] – Password are not hidden in summary installation parameters

Clean up

• [NXP-10842] – inconsistent date formats in advanced search

Improvement

• [NXP-10841] – wrong date value used when querying for documents from the advanced search form

New Feature

• [NXP-10268] – Add a token based authentication module
• [NXP-10306] – Automation client: add an API to get a session using a token

Task

• [NXP-10297] – Ensure test features use FeaturesRunner#getConfig(Class<T> type) instead of FeaturesRunner#getScanner()
• [NXP-10609] – Add internal-snapshots and internal-releases into qapriv profile

* Note that major versions of the Nuxeo Platform are openly available for download. Hot fixes are available for Nuxeo Connect clients as part of their subscription package.

The post [Nuxeo Release] Nuxeo Platform Hotfix 5.6.0-HF10 Is Available appeared first on Nuxeo Blog.

Choose a Nuxeo SDK

Today I’ll write a post a little differently than usual. I won’t write about code per say but about my experience with IntelliJ IDEA. Some of you might be aware that one of our developers has developed an IDEA plugin for Nuxeo, so I decided to give it a try.

First thing I did was import Nuxeo Platform source code using maven. All I had to do was tell IDEA where the root pom and the addon parent pom were. As you know we have many modules so I was expecting this step to take forever. I’m happy to say I was wrong. It went really fast. And I didn’t have to specify my maven repo as an environment variable. What surprised me though is that there is no default JDK, hence there are errors everywher — something fixed quickly using the contextual help. Anyway, it feels great just to navigate in the code. It feels a lot more responsive than the last version of Eclipse, for instance. So, I might seriously consider to switch just on that feeling.

About the Nuxeo plugin: It’s available on the official repository so it’s really easy to install. To configure it, the first thing to do is add the Nuxeo SDK under Settings/Nuxeo/Nuxeo SDKs

Then go back to the main Nuxeo menu and select the newly added SDK. For the hot reload to work, you need to add the sdk template to nuxeo.conf. Simply click on configure; it will add the template for you. The hot reload will then work because the default tomcat valve will be replaced by the one from the template. You’ll find more details on a previous post.

Nuxeo Shell in IntellijIDEA

Now if you hit CTRL+ALT+R, it will trigger hot reload of the bundles having the Nuxeo facet. To test this, I’ve created a very simple bundle with a Listener that logs “Bundle reloaded” each time an event occurs in Nuxeo. To add the facet to that module, go in the project structure, find the module you want to hot reload, right click on it and you should be able to add the Nuxeo facet. Then don’t forget to tick the ‘enable hot reload’ box.

The other cool feature coming with this plugin is the Nuxeo Shell, made available from within IDEA. It allows you to connect to any Nuxeo instance and run scripts or automation chain.

Anyway, as you can see, the IntelliJ IDEA plugin for Nuxeo lacks some features compared to the Eclipse plugin. I might try to add the SDK in the classpath for Nuxeo module as well as the automatic maven dependency management from the SDK in the coming days. Keep in mind that it’s currently not officially maintained by Nuxeo, but feel free to ask questions on Nuxeo Answers or even to contribute to the project :)

Let us know what you think about IntelliJ IDEA or about your choice of IDE while coding with Nuxeo. Come have a chat with us on our Google community page.

The post [Monday Dev Heaven] Trying IntelliJ IDEA and Nuxeo Platform appeared first on Nuxeo Blog.

How to use relative date in nuxeo Studio

Here’s a question about relative date usage in Nuxeo Studio: How do I pass a date, relative to the current date, into a Query Filter?

This is interesting as sometimes you might want to retrieve all documents with an expiration date before today plus 30 days. Or maybe you want to retrieve every document created thirty days ago. You obviously can’t hardcode the date so you have to use something like a @{CurrentDate} object.

Content Automation

If you’re in the content automation context, it’s really straight forward. Most of the operation parameters support the MVEl scripting language. So you have access to any objects like @{CurrentDate}.

Here’s a query that returns the documents that will expire in the next 10 days:

SELECT * FROM Document WHERE dc:expired BETWEEN @{CurrentDate} AND @{CurrentDate.days(10)}

While this retrieves all the documents that have been expired at least 10 days:

SELECT * FROM Document WHERE dc:expired <= @{CurrentDate.days(-10)}

Content Views

Unfortunately, this won’t work in a content view. The problem is that the @{CurrentDate} object (notice the first letter, upper-case C) is only available in the MVEL context. This means that it will work fine in an operation, but it won’t be available as a parameter of a ContentView. Something quite misleading is that there is a #{currentDate} (first letter, lower-case c) object available in the context of a ContentView. It’s outjected by SEAM and is an instance of java.sql.Date, which means you can’t use the days method. There is currently no way to have a relative date with this particular Date object.

So if you want to have a specific parameter, you’ll have to create a seam bean with the appropriate method or factory and use it as a query parameter for your content view. Of course this is not ideal, but we will work on this and try to find a simpler solution.

The post [Q&A Friday] Relative date usage in Nuxeo Studio appeared first on Nuxeo Blog.

Hi, here’s a short entry to tell you we’ve updated the Nuxeo Cookbook with the list of blog posts I’ve been writing for the past year.

You can sort the different posts by clicking on the columns header. They have all been tagged so you can easily search what interests you. All tags are also available on this blog so you can keep navigating through similar posts from the blog. I’ve added a difficulty level to each of them as well as a GitHub repository or Gist when available.

Please give us your feedback. Let us know if the tags are good enough, whether or not it makes it easier for you to search through blog posts, if the difficulty level are correct, etc. We’ll try to make this our main source of how-to and tutorials for all the new developers or for the one wanting to learn new technical Nuxeo-related stuff in general.

The post Nuxeo Cookbook Updated appeared first on Nuxeo Blog.

Our DevOps Julien Carsique recently created a new page on our wiki. It’s a list of the Jenkins plugins we use on our Jenkins instance at Nuxeo, and there are a lot of them. Core plugins (and those that used to be part of the core) are not listed (i.e. Ant, Javadoc, SVN). Let us know what you think about this list, especially if you think we’re missing an amazing plugin! We’re always looking to improve our CI and quality assurance process.

Julien also recently did a webinar on build and systems management. He gave an overview of our build and systems management at Nuxeo, explained which CI and QA tools you can access for your own Nuxeo project and showed how you can borrow ideas from Nuxeo to manage your applications.

You can take a look at his slides on Slideshare or watch the webinar replay right here:

The post Jenkins plugin used at Nuxeo now listed on our wiki appeared first on Nuxeo Blog.

Can an external web service call be included in an automation chain?

Here’s a common question: Can an external web service call be included in an automation chain?

There are many uses cases to this. You may want to integrate with an existing application or simply retrieve some metadata from a single web service. This can be a good way to enrich metadata about a specific kind of document. Let’s say, for instance, that you are managing a book collection in Nuxeo. Every book as an ID called ISBN, which stands for International Standard Book Number. It’s a unique numeric commercial book identifier. Using this ID, you can retrieve more information about the book. The openlibrary.org website has a REST API that gives you information from an ISBN. So, all you have to do to use this in an automation chain is to create an operation that makes a call to this API. As Vlad answered, we have some documentation for building a SOAP-based WebService client in Nuxeo.

Here is an example for a REST client; the complete sample is available on GitHub.

/*
* (C) Copyright 2013 Nuxeo SA (http://nuxeo.com/) and contributors.
*
* All rights reserved. This program and the accompanying materials
* are made available under the terms of the GNU Lesser General Public License
* (LGPL) version 2.1 which accompanies this distribution, and is available at
* http://www.gnu.org/licenses/lgpl.html
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* Contributors:
* ldoguin
*/

package org.nuxeo.isbn;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;

import org.json.JSONException;
import org.json.JSONObject;
import org.nuxeo.ecm.automation.core.Constants;
import org.nuxeo.ecm.automation.core.annotations.Operation;
import org.nuxeo.ecm.automation.core.annotations.OperationMethod;
import org.nuxeo.ecm.automation.core.collectors.DocumentModelCollector;
import org.nuxeo.ecm.core.api.ClientException;
import org.nuxeo.ecm.core.api.DocumentModel;
import org.nuxeo.ecm.core.api.model.PropertyException;

import sun.net.www.protocol.http.HttpURLConnection;

/**
* @author ldoguin
*/
@Operation(id = ISBNQuery.ID, category = Constants.CAT_DOCUMENT, label = "ISBNQuery", description = "")
public class ISBNQuery {

    public static final String ID = "ISBNQuery";

    public static final String QUERY_ISBN_URL = "http://openlibrary.org/api/books?bibkeys=%s&format=json";

    @OperationMethod(collector = DocumentModelCollector.class)
    public DocumentModel run(DocumentModel input) throws MalformedURLException,
            PropertyException, ClientException, JSONException {
        String isbn = input.getProperty("isbn:isbn").getValue(String.class);
        isbn = "ISBN:".concat(isbn);
        String query = String.format(QUERY_ISBN_URL, isbn);
        URL url = new URL(query);
        HttpURLConnection urlConnection = null;
        try {
            urlConnection = (HttpURLConnection) url.openConnection();
            BufferedReader in = new BufferedReader(new InputStreamReader(
                    urlConnection.getInputStream()));
            StringBuffer sb = new StringBuffer();
            String inputLine;
            while ((inputLine = in.readLine()) != null) {
                sb.append(inputLine);
            }
            JSONObject jso = new JSONObject(sb.toString());
            JSONObject metadata = jso.getJSONObject(isbn);
            String bib_key = metadata.getString("bib_key");
            if (bib_key != null) {
                input.setPropertyValue("isbn:bib_key", bib_key);
            }
            String info_url = metadata.getString("info_url");
            if (bib_key != null) {
                input.setPropertyValue("isbn:info_url", info_url);
            }
            String preview = metadata.getString("preview");
            if (bib_key != null) {
                input.setPropertyValue("isbn:preview", preview);
            }
            String preview_url = metadata.getString("preview_url");
            if (bib_key != null) {
                input.setPropertyValue("isbn:preview_url", preview_url);
            }
            String thumbnail_url = metadata.getString("thumbnail_url");
            if (bib_key != null) {
                input.setPropertyValue("isbn:thumbnail_url", thumbnail_url);
            }
            in.close();
        } catch (IOException e) {
            throw new RuntimeException(e);
        } finally {
            if (urlConnection != null) {
                urlConnection.disconnect();
            }
        }

        return input;
    }
}

The post [Q&A Friday] Can an external web service call be included in an automation chain? appeared first on Nuxeo Blog.

As I discovered when debugging TCP connections stuck in the CLOSE_WAIT state for one of our customers, we were using HttpClient incorrectly. We’re not alone in this case, as you’ll find out if you google HttpClient CLOSE_WAIT, but it’s very non-intuitive. Even the official tutorial is wrong, so I’m describing the issue here.

Apache HttpClient is usually used like this in basic mode:

HttpClient httpClient = new HttpClient();
HttpMethod method = new GetMethod(uri);
try {
    int statusCode = httpClient.executeMethod(method);
    byte[] responseBody = method.getResponseBody();
    // ...
    return stuff;
} finally {
    method.releaseConnection();
}

But this is not enough.

The issue is that releasing the connection makes it available again to the HttpClient instance, but does not close it, because HTTP 1.1 is used and it can pipeline further requests to the same host:port in the same connection.

Even though the server may have decided to close its end of the connection, on our client side the connection is still open and will stay that way until an attempt to read from it is made (at which point the client will detect that the other end is closed). TCP works like that, there is a notion of half-closed connection, because close() actually just means I will not send any more data, and you can still receive data from a connection that you closed but which has not yet been closed on the other end.

So what happens then is that when the HttpClient instance goes out of scope it becomes available to the GC, but it will not be garbage collected immediately. Until the GC collects it, the socket connection held internally will stay open and the socket will be stuck in the CLOSE_WAIT state.

To fix this, the simplest way is to add:

method.setRequestHeader("Connection", "close");

before executing the method. This will instruct HttpClient to close the connection by itself once the full response has been received. Another way is to do in the finally block:

httpClient.getHttpConnectionManager().closeIdleConnections(0);

An even better way is to not use a new HttpClient object each time, but to reuse one that has been initialized with a MultiThreadedHttpConnectionManager sized appropriately. Of course in this case the connection manager must be shut down properly when the application shuts down:

private MultiThreadedHttpConnectionManager connectionManager;
private HttpClient httpClient;

public void init() {
    connectionManager = new MultiThreadedHttpConnectionManager()
    // ... configure connectionManager ...
    httpClient = new HttpClient(connectionManager);
}

public void shutdown() {
    connectionManager.shutdown();
}

public String process(String uri) {
    HttpMethod method = new GetMethod(uri);
    try {
        int statusCode = httpClient.executeMethod(method);
        byte[] responseBody = method.getResponseBody();
        // ...
        return stuff;
    } finally {
        method.releaseConnection();
    }
}

Florent

The post Using HttpClient properly to avoid CLOSE_WAIT TCP connections appeared first on Nuxeo Blog.

How do we search for accented characters in Note content?

Here’s a question from patrek; you might have seen him a lot on Nuxeo Answers: How do we search for accented characters in Note content?

If you are not familiar with Nuxeo you have to know that the Note document type is used to store text. This text can be plain text, XML, HTML or markdown. When you store a note in HTML, the accented characters you type are converted to HTML entities. For instance if you type ‘é‘, it will be stored as ‘&eacute;. And if it’s stored as ‘&eacute;‘, it will be indexed as such. Which means that if you search for it, you’ll have to type ‘&eacute;‘ in the search field instead of ‘é‘. This is no good for most of the normal users; they want to type ‘é‘ just like they did when writing the note. So, we need to unescape those HTML entities, but not all of them. This was stored as HTML for a reason. You don’t want to unescape ‘<‘, >‘ or &‘. What we want here is basically to unescape only the ISO-8859-1 characters. A list is available on the w3c reference page. In the old HTML version, it makes sense to have ISO-8859-1 characters as HTML entities as they were not part of the HTML spec. But now, every browser supports those characters so we can safely unescape them.

You have to know that there is already some HTML sanitizing done automatically for the HTML content of notes. Sanitizing is mostly used for security reasons, particularly to avoid XSS vulnerabilities. Take a look at the listener that does this. As you can see, it’s a synchronous listener, triggered very early and only when you’re about to create an event or modify it. What we have to do is unescape those HTML entities right after they’re sanitized. Here’s the appropriate listener contribution:

  <extension target="org.nuxeo.ecm.core.event.EventServiceComponent"
    point="listener">

    <listener name="htmlunescaperlistener" async="false" postCommit="false"
      class="org.nuxeo.sample.HtmlUnescaperListener" order="-5">
      <event>aboutToCreate</event>
      <event>beforeDocumentModification </event>
    </listener>
  </extension>

Now we need the code that does the actual unescaping. The complete sample is available on Github. Here I’m using the Apache StringEscapeUtils.unescapeHtml method. Be careful as it might unescape too much :) Take a look at the unit test and complete it if you want to be sure.

/*
 * (C) Copyright 2013 Nuxeo SA (http://nuxeo.com/) and others.
 *
 * All rights reserved. This program and the accompanying materials
 * are made available under the terms of the GNU Lesser General Public License
 * (LGPL) version 2.1 which accompanies this distribution, and is available at
 * http://www.gnu.org/licenses/lgpl-2.1.html
 *
 * This library is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
 * Lesser General Public License for more details.
 *
 * Contributors:
 *     ldoguin
 */
package org.nuxeo.sample;

import org.apache.commons.lang.StringEscapeUtils;
import org.nuxeo.ecm.core.api.ClientException;
import org.nuxeo.ecm.core.api.DocumentModel;
import org.nuxeo.ecm.core.event.Event;
import org.nuxeo.ecm.core.event.EventContext;
import org.nuxeo.ecm.core.event.EventListener;
import org.nuxeo.ecm.core.event.impl.DocumentEventContext;
import org.nuxeo.ecm.core.schema.FacetNames;

/**
 * @author ldoguin
 */
public class HTMLUnescaperListener implements EventListener {

    public void handleEvent(Event event) throws ClientException {

        EventContext context = event.getContext();
        if (!(context instanceof DocumentEventContext)) {
            return;
        }
        DocumentModel doc = ((DocumentEventContext) context).getSourceDocument();
        if (doc.hasFacet(FacetNames.IMMUTABLE)) {
            return;
        }

        String html = doc.getProperty("note:note").getValue(String.class);
        String unescapedHtml = StringEscapeUtils.unescapeHtml(html);
        doc.setPropertyValue("note:note", unescapedHtml);
    }

}

Here’s the unit test. Note that we are deploying the htmlsanitizer bundle to have a test closer to reality.

/*
 * (C) Copyright 2013 Nuxeo SA (http://nuxeo.com/) and others.
 *
 * All rights reserved. This program and the accompanying materials
 * are made available under the terms of the GNU Lesser General Public License
 * (LGPL) version 2.1 which accompanies this distribution, and is available at
 * http://www.gnu.org/licenses/lgpl-2.1.html
 *
 * This library is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
 * Lesser General Public License for more details.
 *
 * Contributors:
 *     ldoguin
 */
package org.nuxeo.sample.test;

import static org.junit.Assert.assertEquals;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.nuxeo.ecm.core.api.CoreSession;
import org.nuxeo.ecm.core.api.DocumentModel;
import org.nuxeo.ecm.core.test.CoreFeature;
import org.nuxeo.runtime.test.runner.Deploy;
import org.nuxeo.runtime.test.runner.Features;
import org.nuxeo.runtime.test.runner.FeaturesRunner;

import com.google.inject.Inject;

@RunWith(FeaturesRunner.class)
@Features(CoreFeature.class)
@Deploy({ "org.nuxeo.ecm.platform.htmlsanitizer",
        "nuxeo-htmlunescaper-sample" })
public class HTMLUnescaperListenertest {

    public static final String BAD_HTML = "<b>&eacute;foo&agrave;<script>bar</script></b>";

    public static final String SANITIZED_UNESCAPED_HTML = "<b>éfooà</b>";

    @Inject
    CoreSession session;

    @Test
    public void sanitizeNoteHtml() throws Exception {
        DocumentModel doc = session.createDocumentModel("/", "n", "Note");
        doc.setPropertyValue("note", BAD_HTML);
        doc.setPropertyValue("mime_type", "text/html");
        doc = session.createDocument(doc);
        String note = (String) doc.getPropertyValue("note");
        assertEquals(SANITIZED_UNESCAPED_HTML, note);

        session.save();
        doc.setPropertyValue("note", BAD_HTML);
        doc = session.saveDocument(doc);
        note = (String) doc.getPropertyValue("note");
        assertEquals(SANITIZED_UNESCAPED_HTML, note);
    }
}

The post [Q&A Friday] How Do We Search for Accented Characters in Note Content? appeared first on Nuxeo Blog.

Hello everyone! The latest tech report has landed! There is lots of work on DAM right now :)

• Core/VCS Optimizations
• Monitoring Nuxeo
• ESB/SOA
• Automation API
• Contributions
• UX
• Misc
• DAM – Layouts/widgets/actions

Core/VCS Optimizations

Benoit, Stephane and Florent are working on several changes in the Core. Lots of them are already merged onto the main development branch.

JDBC Pool and DataSources Management

DataSources have recently been:

- internalized (declared via a Nuxeo Extension Point system rather than at container level)
- aligned on Tomcat pool (to avoid the leaks issue in Apache DBCP)

There are still some issues; this requires more testing.

Hibernate, Audit, XA and Seam

After a great fight, Stephane is now reaching the goal :

- fixed DataSource management
- updated Hibernate libs
- recompiled Seam with new dependencies (to match hibernate)
- changed H2 config (to have true MVCC via Server mode)
- fixed Fulltext extraction workers

Some cleanup was also started inside the tests, so that we don’t have anymore errors caused H2 concurrency issue. Stephane did the required fixes in most Nuxeo repositories except on AddOns. The goal is to finish the cleanup on nuxeo-features and merge, then :

- do an quick explanation to all dev team
- dispatch the addons tests among developers

UUID Formatting

UUID format change is done for PGSQL (using PGSQL native support for UUID). To go further, we are now switching from UUID to database-generated Long (i.e. sequence). This will have an impact on UI (and potentially security): docs ID will be 0001, 0002, 0003 …

But on the other hand, the DB should be much faster:

- UUID size reduced ( 36 Bytes in String => 16 Bytes in UUID => 8 Bytes in Long )
- processing on Long is better that on String

Anyway, the “Long format” won’t be the default. It may come in handy for very large repositories.
NB: In the near future, we may use Redis to manage ID generation and then migrate Cluster stuff in Redis too.

Ben did the benchmarks for UUID vs VARCHAR.

Reader Benchmark
Writer Benchmark

The results are almost the same when using Sequence instead of native UUID :

+10% / 15% improvement in throughput
response time are 15 to 30% faster

Tag

Florent added a way to query on tags using NXQL.

The documentation now describes how it can be used.

Examples from that doc:

- SELECT * FROM Document WHERE ecm:tag = ‘tag1′
- SELECT * FROM Document WHERE ecm:tag IN (‘tag1′, ‘tag2′)

– documents with either tag
- SELECT * FROM Document WHERE ecm:tag/* = ‘tag1′ AND ecm:tag/* = ‘tag2′

– documents with both tags with queryAndFetch:
- SELECT ecm:tag FROM Document WHERE dc:title = ‘something’
- SELECT ecm:tag FROM Document WHERE ecm:tag LIKE ‘abc%’
- SELECT ecm:tag/*1 FROM Document WHERE ecm:tag/*1 LIKE ‘abc%’ AND ecm:tag/*2 = ‘tag1′

– simpler version of above
- SELECT ecm:tag FROM Document WHERE ecm:tag LIKE ‘abc%’ AND ecm:tag/* = ‘tag1′

Versioning

Based on customer feedback, the versioning policy has been changed when documents are created/updated with the FileManager. Versioning policy can now be configured in the FileManager and it is deactivated by default. These changes impact:

- Import action
- Drag&Drop action

However, this creates a consistency issue with other create/update actions:

- create/update via the Forms
- create/update via Drive
- update file via D&D
- create/update via automation

The proposed solution could be:

- activate FileManager versioning by default in 5.7
- add create version action (check-in minor and check-in major) in the summary tab
- keep the version increment option in edit form
- make update Blob via D&D consistent with FileManager versioning policy

Drive has planned a more clever policy: A version is created if..

- the author has changed
- last update was a long time ago

We need to work on a global spec so that we can be consistent. Any feedback is welcome :)

Monitoring Nuxeo

Current Situation

Today, we have a Counter/Probe system based on Java Simon. Our current solution lacks several aspects:

- JMX/REST API isn’t very friendly (ex: no simple getter via JMX)
- we don’t have any infrastructure to aggregate and correlate data
- we don’t have a set of “smart” counters for monitoring a “standard Nuxeo server”
- we don’t have a monitoring console

Ben and Mathieu started looking at a global solution. Based on their feedback, the couple Metrics+Graphite was selected.

Metrics

Metrics library provides all the primitive features we need:

- counters
- gauges
- meters
- health checks

Ben started plugging some of the existing Nuxeo Counters inside Metrics to be able to test them. See NXP-11079.

Graphite

Graphite is in charge of:

- storing the data history (with configurable archiving policies)
- providing operations on the collected data (derivative, aggregation, …)
- providing graph features

The data for the graph is fetched:

- from Metrics for Nuxeo specific data
- from Diamond for system info
- from Diamond for DB info

Testing!

Ben and Mathieu have setup a Graphite server and defined a configuration to monitor Nuxeo’s server during a benchmark. The goal is to validate:

- the architecture and configuration
- that Metrics does not consume too many resources
- that we indeed have all valuable information exposed

The result of this is:

- a benchmark with all monitoring data
- a pre-version of the documentation about “monitoring Nuxeo”

Ben integrated a Graphite Dashboard on CI. See sample dashboard.

Next steps

Ben will start the merge/cleanup :

- remove Simon
- remove nuxeo-runtime-management (or at least everything that is not useful anymore)
- write a documentation

ESB/SOA

Talend

In the context of EasySOA, a Nuxeo-Talend connector was started: https://github.com/tiry/nuxeo-talend-components
We’re starting to have a basic connector.

Mule

While Tiry was at it, he tried another OpenSource ESB, Mule.
The goal is the same as with Talend, a simple connector. The result can be seen here: https://github.com/tiry/nuxeo-mule-connector

If you wonder what Mule can be used for, take a look at its list of connectors: SalesForce, GitHub, Jira, GoogleDoc, Magento, Jenkins, Redis, SAP, SugarCRM … they are all here!

OAuth 2

From the beginning, OAuth provides mechanisms to manage authentication between two applications, or one user and two applications. But with OAuth1, the system was really not easy to use in a SOA Context.

With OAuth2, the situation is different: Protocol was updated to better fit with the SOA requirements. As a result, OAuth2 is used more and more in SOA and is by default integrated inside Mule and the Connector Framework.

Content Automation and SOA

Content Automation and Code Generation

With both Talend and Mule, we had the same issue: Having a list of operations that is dynamic is not compliant with ESB standard logic. Most systems seem to assert that the list of operation is known. This seems indeed to be true for most systems, but not for content automation, which is one of its main strengths.

In the context of Mule, it is very easy to expose automation operation via UI or even via SOAP, if I have the java signature. Based on our current model, the idea would be to generate a class that contains all the java signatures of accessible operations, and then let Mule DevKit generate all the plumbing code.

Work on Complex Types

All work done is inside a branch. Next steps:

- validate non-regression with Ben and merge
- finish processing of Complex Types

Automation Schema Settings

In current automation implementation, the choice of schemas/properties that must be fetched on documents before marshaling is defined by the Http Header X-NXDocumentProperties. While it does work, there is no global setting to define default configuration in the Java Automation client. Since this would be useful for the Mule connector, this will be added.

Automation API

As you know, content automation is a corner stone of the Nuxeo Platform:

- a lot of clients are using Nuxeo remotely via automation
- Automation is the power of Studio
- REST API rocks

Swagger like API Doc

I’ve started playing with REST API Documentation solutions like Swagger and Apiary; you can see a preview here.

The goal is to have:

- a nice Rest API documentation
- a JS based console to be able to easily test API

We’ll see if we can leverage Swagger/Apiary code + some Lise CSS wizardry to make our automation API look nicer and provide a real JS based console.

NXP-11095 was created to track this work on API showroom.

Exception Handler/Debug

In the current model, it can be very complicated to handle exceptions inside an automation chain. The same pain applies when you want to understand why a chain is crashing. This is clearly something we must improve, because with workflow, we have more and more complex chains.

Exception Management

When running an automation chain, errors can occur (ex: mail server not available, doc. not found …). We must provide a way for Studio users to be able to define how exceptions must be managed. Based on what we saw with ESB, a solution is to define a chain dedicated to exception management. You can take a look at Mule documentation for an explanation of this model.

Trace

Stéphane started some work for support about capturing automation context to help debugging: see NXP-10081. It is basically about stacking context, and input and output for each operation that is executed, so that in case of error we can track what happened.

This unfinished work is, for now, in a branch on automation-core. This tracing system should be activated:

- when dev mode is activated (or may be a dedicated flag as suggested by Florent)
- when an exception management chain is defined

Complex Types and Branch Merge

The refactoring on marshaling is still waiting in his branch NXP-10080.

Before finishing and merging this work, we should check that we won’t break the protocol: see NXP-11093.

Tiry started doing http capture/replay, but Julien may have a solution to run tests with nuxeo-automation-client V(n-1) against nuxeo-automation-server V(n).

Automation Chain Parameters

Currently we have no easy/clear way to define what the parameters of a chain are. In fact, currently a chain can only retrieve values from the automation context. See NXP-11096 for details.

Contributions

OAuth 2 and OpenID

Nelson contributed some code to support OAuth2 and OpenID. This code was partially merged in a branch:

- for adding OAuth2 service providers
- for adding support of OpenID

This is just the first step since a lot of things should be added to have a real OAuth2 support:

- Nuxeo should be accessible as an OAuth2 provider (i.e. call Nuxeo via OAuth2)
- Nuxeo Opensocial should be able to use OAuth2

But before going further, we need to solve some issues:

- Google OAuth2 Libs depends on Guava
- Shindig and GWT depends on google-collections

Guava is actually a new version of google-collection, but the compatibility does suck.
As a result:

- with Guava, Nuxeo does not start since Shindig startup failed
- with Google Collection, OAuth2 startup fails

Upgrading to Guava implies upgrading Shindig …

This explains why:

- the OAuth2 partial support is not committed in master
- I needed a to use a homebrew Google Collection to be able to test OpenID. See NXP-10907.

UX

Log-in Screen

For the log-in screen, we have several aspects we would like to make configurable:

- some of the CSS properties
- define visibility of some page blocks (like for the right banner, header, footer … )
- add OpenID providers icons/links

Doing this should solve at least two problems:

- Studio customization of the log-in screen
- OpenID integration

Basically, the approach will be to integrate inside the authentication service a new extension point that provides plugability for OpenID and Studio requirements; see NXP-10918.

Safe Edit

Safe Edit mode NXP-10258 is almost ready, and will be available in the next days:

- add a WebDriver test
- adjust SafeEdit/Page leave detection
- activate it in the trunk

Access Keys/Ajax

We already have the “infrastructure” to associate key binding to actions. See NXP-9383.
We still need to figure out:

- What are the actions that should be associated with shortcuts
- How we advertise shortcuts

Admin Message

Lise will move the “admin message” to bottom of the page.

WAR Resources Cleanup

Lise wants to start a cleanup work on the icons. There are several goals:

- remove icons that are not used
- change the names that don’t make sense
- switch to PNG when needed

Before we start this work, we should define the scope of the changes and review compatibility impacts. The idea is that Lise will start a webapp-resources-compat that will hold resources that have been renamed or deleted from the trunk version.

The bundle could also hold other deprecated web resources: xhtml, js… We surely already have some of those…

Misc

Web Mobile

The mobile webapp has been aligned on Cordova 2.2, but since the project is moving fast we are still late! Anyway, the build and packaging has been automated for both iOS and Android and is now part of our CI chain (using a MacOS Jenkins slave!). The CI chain produces the binaries:

- APK can be installed on any Android device that is configured to accept dev applications
- iOS app can be installed on devices that have been explicitly listed in the build config file

Automating build and deployment on an Android emulator is not an issue, but WebDriver and Cordova both instrument the Android WebView so WebDriver and cannot be run against a Cordova enabled WebView …

Current features include:

- browse content
- download files
- import file
- file exchange between Nuxeo App and other apps on the device

Quota

Mariana has taken over the dev of the quota add-on. What has been done:

- better UI (including JS charts)
- better enforcement of rules (check quota consistency over the tree)
- better management of versions

What she’s working on:

- probably some UI changes (based on Lise’s feedback)
- admin center part
- reporting on quota usage (simple content view)
- manage max quota on personal workspace
- initial computation: add support for versions

For version management, we may want to add a configuration to know how we count versions:

- count size of version Blobs (what is done currently)
- only count Blobs that are different from live doc (much more complex because of the possibilities to delete/restore versions)

DAM – Layouts/Widgets/Actions

Some of the work on DAM is the opportunity to:

- make DAM completely configurable via Studio
- improve our layout/widget infrastructure

NXROADMAP-131
NXROADMAP-127

What Was Recently Done

FancyBox

Fancybox based actions now support forms inside their included popup, see NXP-9444. That makes it possible to change the ‘new’, ‘import file’ actions into real actions ([NXP-9433], still needs a bit of styling) as well as the ‘new asset’ action in DAM.

This makes the model clean and this will allow us to use it inside Studio. This work will be re-used to call an automation chain via a form (also inside workflows).

PageProvider/Refresh and Events

DAM may require some changes in the PageProvider caching/invalidation system.
There is no hurry since it works for now, but:

- it means that we should improve the framework
- it would be cool that content view refresh does work in DAM

Changes have been integrated into master (NXP-10851).

NB: implied to do changes in most of the PageProviders

URL Codec

DAM requires some changes in URL Codec, basically because the URL now stores information about the content view (which page …). These changes, as well as the URL client side update could be pushed to CAP/DM. Thomas and Anahide will review that. This could open the way to switch CAP/DM in Ajax too.

The goal is to be able to generate a URL for a content view that can be bookmarked + use Html5 history.push (see https://jira.nuxeo.com/browse/NXP-10978).

In Progress

Thumbs View

The thumbs view (default view) in DAM was rebuilt using a thumbnail adapter and a layout(NXP-10825).

The DM icon listing view should be aligned on the same infrastructure. In addition, we must ensure that Studio has the ability to generate such a layout. For that, we need to build a box-like listing layout template.

Bulk Edit

Arnaud and Thomas are working on NXP-11054 to reenable Bulk Edit in DAM and at the same time integrate it inside DM. For now the scope is a “Simple” bulk edit using a pre-configured layout (`damBulkEdit` in DAM, `bulkEdit` in DM). The layout will be able to be overridden through Studio.

NB: Based on what was done with diff, Antoine may have ideas for dynamically computing a “shared layout”

Saved Search

Vlad is working on NXP-11056 to build a content view save feature. Code will be based on the faceted search save code.

restDocumentLink

Make restDocumentLink generate URL with a repositoryName instead of a Document: NXP-11045

What’s in Progress

Content View Configuration in Studio

Make content view slots more configurable, including from Studio:

- extract selection related actions in a separate form
- add configurable page size range

ReRender – Ajax Panel Identification

DAM uses Ajax; this means the widgets should reference the zones that should be refreshed. In the future we may want to extend the ID generation system to be able to handle the Ajax refresh use case as well as the cross validation issues.

But for now, we’ll look at something simpler and faster. The idea is to add to the container widget an attribute to be able to force the ID associated to the nested a4j:panel element. For DAM, this should be enough; we only need three zones and this would make DAM configuration simpler.

Define container widget holding an outputPanel and allowing to set the ID (NXP-11029).

As decided, DAM defines fixed/hardcoded refresh zones. Configuring Ajax behavior involves actions and widgets:

- the action defines:
- if Ajax is supported
- what are the rerendered zones

- the widget that renders the actions defines:
- if Ajax is used
- an optional rerender zone

NB: adding more Ajax in the pages may force us to sync all Ajax queue to an unique queue.

Widgets that Render Actions that Define Widgets…

In the current model, the action categories are handled by different templates that are aggregated in the global template used by the widget. To make this completely pluggable, we could use WidgetTypes to define rendering of Action Category(NXP-11026).

Documentation!!!

The action/widget model was greatly improved to support new use cases. As a side effect, the model is more complex: more options, more opportunities to make mistakes…

Tasks for updating documentation are already created. But because the model (and vocabulary) is complex, we should leverage layoutDemo to demonstrate use cases for Ajax refresh and widgets/actions composition.

Actions EL Rationalization

We should use JSF EL for action filters. The current action filter implementation relies on a bad JEXL integration. Aligning on JSF EL will:

- make context and syntax more consistent between actions and widgets
- fix bugs in the current action filter system

Since we are using more and more actions and are now also mixing them with widgets, it really makes sense to address this point.

About Actions

The asset browser view (as the summary view in DM) is based on a grid layout. This grid layout contains three zones, each containing an action widget. In each of the three zones, the action widget will render subwidgets defined by actions. Using the action system rather that simply nesting subwidgets has several advantages:

- it’s easy to manage visibility via filter
- it’s easy to manage ordering
- it’s easy to contribute a new subwidget from an add-on

This is good, but the sad point is that this is not how Studio Layout Editor works. Indeed Studio editor only defines nested widgets via a “WYSIWYG” editor.

As a side effect, when changing the summary tab of DM:

- you have to rebuild it from the ground up
- you potentially lose all the widgets contributed by add-ons

This is not ideal, and this is probably not what we want for DAM. Basically the real problem is that:

- Studio’s editor manages widgets in a static way
- Studio can not easily do widget rendering based on filters

A possible option would be to have an hybrid model:

- add action attributes at the widget level
- so widgets can be placed in Studio normally
- widgets have extended properties (controller) such as filter, category…
- dynamically generate the required actions at widget deployment time
- so that Studio does not have to be aware of actions

Thomas and Anahide will work on this in the next few weeks and this may create new features inside Studio:

- View editor (not a layout, not a tab, but a view!)
- Global widget definition (to allow reuse)

Logically, this brings back the question about what is a target platform and how we will define the default config for DAM actions/widgets.

The post [Nuxeo Tech Report] News from the Developer Front #4 appeared first on Nuxeo Blog.

Automatic creation of user workspaces

Today we’re answering a question about user workspaces. How can you create them automatically? First, if you don’t know what a personal workspace is, here’s a definition taken from our documentation:

Personal workspaces can only be accessed by their owner, by default. You can of course share the access to your personal workspace with other users. Personal workspaces are accessible in the header of the application.

What you need to know to answer that is that those personal workspaces are usually created lazily. It means they are created the first time something tries to access it. So the solution Ron proposed is to access the user personal workspace each time he successfully logged in.

You can access it using the following Java code:

  UserWorkspaceService userWorkspaceService = Framework.getLocalService(UserWorkspaceService.class);
  userWorkspaceService.getCurrentUserPersonalWorkspace();

But, to do that, everybody would have to log in. If you still need to create all of them automatically, the good news is that there is a new method that takes a detached user as input. This means you can call it for every user of your application. But, as you can see, it’s really new and not released yet. It will be available on Nuxeo 5.7.

    /**
     * Gets a detached user workspace of a specified user depending of the
     * passed principal.
     *
     * @param principal of the wanted user's workspace owner
     * @param context is a document to determine the domain
     * @return the DocumentModel for the personal workspace
     * @since 5.7
     */
    DocumentModel getUserPersonalWorkspace(NuxeoPrincipal principal,
            DocumentModel context) throws ClientException;

So you could use this in an operation like this:

	@Context
	UserManager userManager;

	@Context
	UserWorkspaceService uws;

	@OperationMethod(collector = DocumentModelCollector.class)
	public DocumentModel run(DocumentModel input) throws ClientException {
		List<String> userIds = userManager.getUserIds();
		for (String userId : userIds) {
			NuxeoPrincipal user = userManager.getPrincipal(userId);
			uws.getUserPersonalWorkspace(user, input);
		}
		return input;
	}

Please keep in mind that this can be a very costly operation. You should probably batch this if you have many users too.

The post [Q&A Friday] Automatic Creation of User Workspaces appeared first on Nuxeo Blog.

You might already know Laurent Dreuillat from a previous interview we conducted after Nuxeo World 2011, where we mostly talked about the customers and projects he encountered. Today’s interview will be more focused on his new job and the recent contribution of their Outlook addon.

So Laurent, the last time we talked with you, you were already working in the ECM field but in another company. You’ve recently founded a new company called Astone Solutions, a Nuxeo Gold Partner. Tell us about Astone Solutions. What are the goals of this new company?

Yes, we decided to create a spin-off from Sword that contains, among others, the ECM part of the Western France Business Unit of Sword. Our primary goal is to provide a concrete solution for the comprehensive management of documents and information within the company and its ecosystem. Astone Solutions is an IT services company specialized in the management and development of business flow. Our core knowledge includes digitization, ECM (with Nuxeo of course ;) ), portals, search, BPM and archiving.

What types of projects have you worked on recently? Is there one that really strikes you as a good use of technology to make business processes more efficient?

Today we are working on a very interesting project for a financial company to migrate their ECM system from a proprietary ECM to Nuxeo. This will represent a huge amount of documents and it will be accessed through a Liferay Portal. For that project, we developed a framework to have better integration of Nuxeo inside Liferay. The goal is to improve the business process by relying on a strong ECM core as well as having a central point for accessing documents and a lot of other functionalities (BPM).

You recently contributed an Outlook addon for Nuxeo that is now available on Nuxeo’s Marketplace. Can you tell us why you developed this addon and what it does?

We developed this addon because a lot of our customers were looking for such a tool. The goal of that plugin is to allow clients to store emails in Nuxeo. This plugin is fully integrated inside MS Outlook (tested on 2003/2007 and 2010). On Outlook, you can select one or more emails, choose dynamically where you want to store them inside the Nuxeo hierarchy and send them to Nuxeo. Each email will be stored inside a mail object containing:

- The original EML file as the main file
- The attached files of the email as attachments
- The main metadata of the email automatically populated (title, subject, sender, receiver, date, …)

In Outlook 2010 you can even select an email attachment directly and send it to Nuxeo as a file object.

Developing this addon required a much different approach than typical configuration projects on the Nuxeo Platform. How deep was your technical understanding of the platform before you developed the addon? Would you say it was easy to develop?

We already worked on other .Net integrations in our previous projects; for example: an integration between Microsoft SharePoint and Nuxeo. It was very easy to develop by using REST operations. The next step would be to create an automation client API for .Net. That would be really great to improve integration between .Net and Nuxeo. Perhaps here’s another opportunity for Astone Solutions to contribute to Nuxeo.

As a systems integrator, you make choices about the platforms or products that you develop and deploy for your customers. Can you tell us why you’ve chosen to work with Nuxeo?

We chose Nuxeo because, for us, it was the best choice in terms of architecture and extensibility. The learning curve may be a bit higher than other products, but the result is definitely better (better quality/scalability). The other very important point is the availability of Nuxeo Studio, which is a very effective tool that helps us save time on projects.

I understand that you also have other projects going on. Is there anything interesting that you plan to contribute, like you did with the Outlook addon?

Yes, we are working on an addon that we could call “Extended LiveEdit”. With that plugin, you will be able to create a new document (the same way as LiveEdit) and to select the type of Nuxeo document you want to create (not just a file object). It will give you the opportunity to select the type of document you can create (based on your security level) and it will dynamically display the creation layout of the document type. You will then be able to fill in the fields (validation rules are included) and validate the document to create it in Nuxeo. This addon works only on MS Office and will be delivered in the Nuxeo Marketplace when it is completely finished.

By Trizek (Own work) [GFDL or CC-BY-SA-3.0-2.5-2.0-1.0],via Wikimedia Commons

- First, I wish that the Nuxeo Platform would continue to evolve at the same pace, and keep this high level of quality.
- Then, I wish to be able to use Nuxeo Drive very soon (very exciting functionality).
- Finally, I wish for the integration of something like CIFS to replace the WebDAV protocol.

Astone Solutions is located in the beautiful region of Brittany, France. Do you have any advice for the travelers in the area?

You should take a trip to the beautiful town of Rennes, then go to the coast to visit Saint Malo, Dinard and Cancale (especially if you like oysters). There are many things to discover in this region.

What do you like to do in your spare time?

In my spare time I like developing new Java frameworks and plugins… No, I’m just joking. As I said previously, I like to travel in Brittany to discover new places and, of course, to eat “Galette Saucisses” (a specialty of Rennes) and drink beer ;)

The post [Nuxeo Community] Meet Laurent Dreuillat appeared first on Nuxeo Blogs.

Our next bug day is coming up next Monday, March 18. If you want to contribute, there are many ways to do so!
For instance, if you have found a bug, and did not find a Jira ticket mentioning it, feel free to open one. The bugs related to Nuxeo Platform are to be filed under NXP, and the bugs related to Nuxeo Studio, NXS. All of the bugs we’ll be working on are tagged with ‘bugday‘.

The tickets that we want to include in the bugday backlog must be:
- Bugs (obviously)
- Quite short to fix (3 hours estimate max)
- User-visible

Now, if you’re a developer and you want to help, this is a nice opportunity to give back to the Nuxeo community. You can take tickets tagged with ‘bugday’ and then submit the fix using GitHub’s pull request. If you’re experiencing difficulties while fixing bugs, you can ping us on our IRC channel (#nuxeo on Freenode). You can also join us on our Google plus community website :)

And, if you are in Paris, you can join us at the office and enjoy a post bug day beer :D

The post Next Nuxeo Bug Day: Monday, March 18 appeared first on Nuxeo Blogs.

How to retrieve custom metadata with Content Automation

Here’s a question that comes up often: When you retrieve a document using content automation, how do you retrieve custom metadata? This question comes upoften mostly because content automation is the preferred way of interacting with Nuxeo from another application. People use it on mobile, directly from Javascript, from another Java app, the PHP client, the python client (Nuxeo Drive is a good example of that), etc.

Now back to the question, how do you retrieve other metadata? The question was about the Android connector but it applies to content automation. The short answer: We have special headers. Here’s the definition straight from the documentation:

This header can be used whenever a document will be returned by the server. The header is forcing the server to fill the returned document up with data from schemas that match the X-NXDocumentProperties filter. So, X-NXDocumentProperties is a filter of schemas. If you don’t use the header, only the minimum amount of required document properties are returned. To have more properties in the returned document, you can specify a list of document schema names:

X-NXDocumentProperties: dublincore, file

or to have all the document content, you can use the ‘*’ character as the filter value:

X-NXDocumentProperties: *

If you are not familiar with Nuxeo, schemas are used to define lists of metadata.

This is intended for developers who build their own requests. If you’re using a content automation client, there’s a good chance this is already part of the client API. Here’s an example with the Java client:

import org.nuxeo.ecm.automation.client.Constants;
import org.nuxeo.ecm.automation.client.adapters.DocumentService;
import org.nuxeo.ecm.automation.client.model.Documents;
Documents docs = (Documents) session.newRequest(DocumentService.GetDocumentChildren)
        .setInput(doc)
        .setHeader(Constants.HEADER_NX_SCHEMAS, "*").execute();

The post [Q&A Friday] How to Retrieve Custom Metadata with Content Automation appeared first on Nuxeo Blogs.

I’m very pleased to tell you that we now have a Kerberos authentication module thanks to Sylvain Chambon. He works at Open Wide, one of Nuxeo’s Gold partners, as Senior Software Architect and Team Manager. You might recognize Open Wide as we recently did a webinar together about EasySOA, a research project Nuxeo and Open Wide have in common.

This feature has been asked for many times by many different people. The Jira tickets go back to 2007. As you can see it’s been a long time :) If you are not familiar with Kerberos, here’s a definition taken straight from the MIT page:

Kerberos is a network authentication protocol. It is designed to provide strong authentication for client/server applications by using secret-key cryptography. A free implementation of this protocol is available from the Massachusetts Institute of Technology. Kerberos is available in many commercial products as well.

A Nuxeo Marketplace package is now available on the dev channel, which is to say it’s only available from the update center of a 5.7-SNAPSHOT version of Nuxeo. The public Marketplace package will be released at the same time as the next 5.6 hot fix. It installs the plugin and configuration files. Be aware that it will not work as is and requires some configuration, which is quite usual with Kerberos setup. Please take a close look at our documentation.

The post New Contribution: Kerberos Authentication Plugin by Sylvain Chambon appeared first on Nuxeo Blogs.

The March 2013 Bug Day is over. On Bug Day, we set aside a full day where we all work together to clean up and fix as many bugs as possible. We have squashed 54 of them with some help from the community. Again, kudos to Nelson Silva who helped us with some pull requests. Here’s a little chart so you can get an idea of the work done. You can also get the full details on our Jira.

A lot of those bugs have been back-ported to the 5.6 branch so you can expect a very efficient hot fix pretty soon. This will really benefit the upcoming 5.7 release. Here’s the list of corrected bugs. Not bad for a day’s work.

The post March 2013 Bug Day Results appeared first on Nuxeo Blogs.

Retrieve the list of all possible subjects values using Content Automation

Today we have a question from thexman who asks how he can retrieve the list of all possible subject values with content automation. His first thought was to use NXQL, Nuxeo’s query language, which makes perfect sense when you are searching for something. Unfortunately, NXQL is used to search the content of documents, and all the subject values are not stored as documents. The values proposed by Nuxeo when choosing a subject are all stored on what we call a directory. Here’s a definition taken straight from our documentation:

In Nuxeo EP, a directory is a source of (mostly) table-like data that lives outside of the VCS document storage database. A directory is typically a connection to an external data source that is also accessed by processes other than Nuxeo EP itself (therefore allowing shared management and usage).

A vocabulary is a specialized directory with only a few important columns that are used by Nuxeo EP to display things like menus and selection lists.

There is a specific operation to retrieve the entries of a vocabulary called Directory.Entries.

Get the entries of a directory. This returns a blob containing a serialized JSON array. The input document, if specified, is used as a context for a potential local configuration of the directory.

Here’s a call example using cURL:

curl -H 'Content-Type:application/json+nxrequest' -X POST \
  -d '{"params":{"directoryName":"topic"}}' \
  -u Administrator:Administrator http://localhost:8080/nuxeo/site/automation/Directory.Entries

This automation call returns the following JSON array on a default Nuxeo instance:

[ { "id" : "art",
    "label" : "label.directories.topic.art",
    "obsolete" : 0,
    "ordering" : 10000000
  },
  { "id" : "human sciences",
    "label" : "label.directories.topic.humanscience",
    "obsolete" : 0,
    "ordering" : 10000000
  },
  { "id" : "society",
    "label" : "label.directories.topic.society",
    "obsolete" : 0,
    "ordering" : 10000000
  },
  { "id" : "daily life",
    "label" : "label.directories.topic.dailylife",
    "obsolete" : 0,
    "ordering" : 10000000
  },
  { "id" : "technology",
    "label" : "label.directories.topic.technology",
    "obsolete" : 0,
    "ordering" : 10000000
  }
]

The post [Q&A Friday] How To Retrieve the List of All Possible Subject Values Using Content Automation appeared first on Nuxeo Blogs.

I’m really pleased to tell you that Nuxeo now supports OpenId authentication, all thanks to our good friend Nelson Silva. Remember him? It’s one of the many things he’s helped us do.

What is OpenID authentication?

If you are not familiar with OpenID:

OpenID is a decentralized authentication protocol that makes it easy for people to sign up and access web accounts.

From a Nuxeo user point of view, it lets you log in with an existing OpenID provider account. No password needed, just make sure the user exists in Nuxeo having a common email address with your OpenID provider. Here’s an example:

I have an existing Nuxeo user who’s email is randomdude@gmail.com. Gmail is an OpenID provider. I’ve configured Nuxeo to use it as an OpenID provider. When I am on the login page, I now have a new choice. It’s a sign in with Gmail button. It will take me to Gmail which will ask me if I want to sign in to Nuxeo, or to sign in to Gmail if I have not already and then sign in to Nuxeo. And that’s it, you’re in. No username/password needed.

Configure a New OpenID Provider

As usual, there is a dedicated extension point. Here’s the default configuration for Gmail from the OpenID addon.

<?xml version="1.0"?>
<component name="org.nuxeo.ecm.platform.oauth2.openid.google" version="1.0">
  <requires>org.nuxeo.ecm.platform.oauth2.openid.OpenIDConnectProviderRegistry</requires>
  <extension
    target="org.nuxeo.ecm.platform.oauth2.openid.OpenIDConnectProviderRegistry"
    point="providers">
    <provider>
      <name>GoogleOpenIDConnect</name>
      <label>Google</label>
      <description>Login using your Google account</description>
      <authorizationServerURL>https://accounts.google.com/o/oauth2/auth</authorizationServerURL>
      <tokenServerURL>https://accounts.google.com/o/oauth2/token</tokenServerURL>
      <userInfoURL>https://www.googleapis.com/oauth2/v1/tokeninfo</userInfoURL>
      <scope>https://www.googleapis.com/auth/userinfo.email</scope>
      <icon>/icons/google.png</icon>
    </provider>
  </extension>
</component>

This configuration is incomplete as you still need to specify the clientID and the clientSecret. They are given to you by your OpenID provider (in this case Google) when you register an application through its developer console, which means you first need to declare your Nuxeo Web Application to Google. To do that, go to https://code.google.com/apis/console, then click on API Access, Create and Web Application. This pair of shared clientID/clientSecret is needed to make sure that your Nuxeo server and your OpenID provider (ex. Google) can trust each other.

So, once you have the clientID/clientSecret, create `nxserver/config/openid-config.xml`with the following content:

    <?xml version="1.0"?>
    <component name="org.nuxeo.ecm.platform.oauth2.openid.google.testing" version="1.0">
      <require>org.nuxeo.ecm.platform.oauth2.openid.google</require>
      <extension target="org.nuxeo.ecm.platform.oauth2.openid.OpenIDConnectProviderRegistry" point="providers">
       <provider>
        <name>GoogleOpenIDConnect</name>
        <clientId><!--enter your clientId here --></clientId>
        <clientSecret><!--enter your clientSecret key here --></clientSecret>
       </provider>
      </extension>
    </component>

The post New Contribution: OpenId Authentication Plugin by Nelson Silva appeared first on Nuxeo Blogs.

Here’s a brand new tech report taken straight from the developer meetings.

• High Volume Infrastructure
• Publishing, Relations, Tags, Rendition, Versions,…
• Connect/Studio
• Automation
• Metrics/Graphite
• Infrastructure Work
• Javascript in Nuxeo

High Volume Infrastructure

Redis and Workmanager

Persistent WorkManager

Ben started using Redis to manage work queues.

The goal is to integrate Redis and the WorkManager so that:

• async jobs can be shared between cluster nodes
• work queues can be mono-threaded, even in a cluster environment
• work queues can be persisted (and not lost at server shutdown)

For this to work for real we probably need to write a custom Redis WorkManager. You can follow the status of this task with NXROADMAP-137 – Provide a distributed workers infrastructure .

Nuxeo and Redis

The support for a persistent WorkManager should be an option: It will depend on Redis (that does not run ‘easily’ under Windows). When it is not activated, jobs are only defined in memory. To make it safer, we can at least warn before restarting the server: ‘your server is currently processing xxx jobs and cannot be shut down right now…’.

We want to be able to use Redis for other use cases (like locks), so we need to share a nuxeo-redis-service that at least provides the configuration for connecting to Redis from Nuxeo.

NB: Ben will create the required Jira issues and associate to NXROADMAP-137

WorkManager and batch mode

In addition, recent benchmarks show that with the current WorkManager implementation we have an issue with mass import. It was working before the latest changes on threadpools, but currently the bench fails. Ben is investigating this point: first goal being to know if this is a test setup issue or a real code issue.

VCS / Soft Delete

Florent started to implement the Soft delete support in VCS. It means doing the deletes in 2 steps:

• mark the deleted objects
• delete marked objects in batch

This approach that avoids “on delete cascade” seems to be better for performance. This will also avoid consistency issues when docs are deleted by synchronous listeners but used by async listeners.

Florent started implementing this and added a new config in the repository descriptor. The code is almost ready for a first test. Once we have validated the principle, we can address the remaining tasks:
- batch delete support
- GC system

Publishing, Relations, Tags, Renditions, Versions

The publishing feature can be improved to address a few issues:

• publish may duplicate some relations
• tags are relations, but on versions, so they are not translated the right way
• rendition publishing overrides the proxy (and then removes relations)

The issue is that we need to work on a specification about:

• what should Nuxeo do
• what is a missing feature
• what is a bug
• what is an evolution

Keeping in mind that this may all be a question of use case.

• some people may want to keep/translate the tags when publishing (ex: using tag as a meta-data)
• some other may want to reset the tags when publishing (ex: using tag as a validation flag)

Feedback and use cases are very welcome :)

Connect/Studio

Next Generation Connect UI

We are working on a new interface for the Nuxeo Connect portal. An AngularJS based web UI for Connect “self-care” is starting to be usable. The associated roadmap task is NXROADMAP-148

Studio Release Schedule

2.10.2

Connect 1.14.1 and Studio 2.10.2 are now up.

2.11 / DAM:

The main focus is to add more DAM features (target is May).

2.12:

This is, at least for now, the Nuxeo 5.7 scope:

• work on Workflows
• work on Action + Filters
• work on Directory Widget
• fix technical debt

Automation

We have started organizing the work on automation. This work include several aspects:

• fix several bugs / limitations: NXP-10078
• REST / CRUD / object mapping: NXP-11284
• Debugging and error management: NXP-11281
• Test, non regression and client validation: NXP-11290

We will also try to ensure convergence of other existing automation clients:

• Python client used by Drive will be managed by Antoine
• Objective C iOS client: see with Arnaud, Ineat and may be Smart&Soft
• Android Client (there is some code we should reintegrate)

Vlad will be the lead developer on the automation topic, but he will get help from several people, especially:

• Julien: on backward compat testing
• Stephane: on debugging
• Olivier: on marshaling
• Damien: on REST binding

The roadmap and NXP Jiras have been updated accordingly: see NXROADMAP-139

Automation debug and error management

There are actually three different use cases we would like to address:

• allow some exception bubbles to the client: ‘get more than Operation XXX failed’
• provide a way to manage exceptions at the chain level: ‘try to execute this operation and otherwise execute an other one…’
• provide a way to debug an automation chain: ‘see what was inside the context when operation X was called…’

Even if these three topics are related, technically the will require separated tasks.

See NXP-11281

Bubble exceptions to client

The idea is that someexceptions should be sent to the client without being wrapped because they contain a message that can be meaningful to the client. We can have two approaches for that:

• use the RecoverableClientException that was introduced for Quota system
• use explicit exception declaration in operation method signature

cf NXP-11283

Automation chain debug

Stephane started some work for support about capturing automation context to help debugging.

Jira task was updated: NXP-10081

Automation and REST binding

cf NXP-11284

Why is automation not RESTful ?

Automation is more a RPC system than a Rest binding. It does actually use JAXRS, but the Rest resource is the operation itself:

• calling GET will give you the definition
• calling POST will execute it
• PUT would create an operation
• DELETE would remove an operation from the server

Why is it an issue ?

Having a simple RESTful CRUD API would help:

• simple testing
• integration with Javascript MVC model

Build a CRUD RESTful API on document and adapter

Here we are talking about reverting the automation paradigm:

• the resource is the document and not the operation
• the ‘Operation’ is the Http Verb and is limited to CRUD

We could build that as an extension of the existing content automation API. This basically means that automation exposes three endpoints:

• Operation RPC endpoint (/site/automation/${operationId})
• Operation RPC ‘batch endpoint’ (/site/automation/batch)
• CRUD (/site/automation/CRUD/${resource})

This new CRUD API should focus on CRUD on a DocumentModel or it’s adapter. Contrary to operations, CRUD API cannot be combined inside an automation chain and doesn’t need to appear inside Studio.

NB: This could also be a good opportunity to add the Blob Download API inside this Rest binding

Metrics/Graphite

Status

Integration is merged in 5.7-SNAPSHOT. Cleanup still needs to be done:

• remove Simon related stuff
• remove Probes (should be replaced by Metrics Health Check)

Documentations and configuration

A draft documentation has been added. Need to add setup documentation NXP-11164. Next bench will show us if we need to add more stuff, either inside Nuxeo (counters) or inside Graphite.

Infrastructure work

Tomcat DS / Hibernate upgrade / Seam patch / H2 upgrade + MVCC branch

Reminder

This branch was created a few weeks ago by Stephane to fix an issue with some DB connections being used in auto-commit mode. Starting from this problem, we discovered linked issues that forced us to do several changes. This new branch contains:

• usage of Tomcat native pools
• an upgrade on Hibernate / Seam
• an upgrade of H2 database
• a switch to H2 MVCC mode

Testing this new branch

On this new branch there are a lot of failed tests: both unit and integration. When looking at the logs it is not clear:

• what error are actually test setup errors
• what are “normal errors”
• what are real errors that actually cause the test to fail

To address that Stephane started a cleanup of the unit tests:

• align all tests on feature runner
  • even JUnit3 tests based on RepositoryTestCase can actually use the FeatureRunner
• allow usage of a feature to capture the log
  • make test capture the log to check for errors
  • make test fail if there are unprocessed / unexpected errors
• fix tests setup / tear down steps
• fix tests one by one

So far this work is not finished since there are still issues with some addons.

Tomcat 7 upgrade

Reminder

The work was started in December in a branch associated with NXP-10071. The branch was not merged because of impact inside addons regarding server-api changes…

How we finally merge

Julien will:

• restart a build
• identify the errors
• create associated Jira tickets

and then we can dispatch the tickets so that we can finally do this upgrade!

JavaScript in Nuxeo

Context

JS is (was?) hype and we’re a little late joining the battle. Let’s get to it.

Javascript issues

Javascript comes with a set of issues:

• poor maintainability
  • JS can be cryptic, (and jQuery does not help)
  • refactoring is a challenge
• no clear dev model
  • prototype base model + several OOP libs
  • JS let’s you do the same things in 3 different ways
  • JS has some built-in flows (scoping, bad closure…)
• no dependency management
• poor testability (at least for now)

All of these points are even stronger when put in the context of a platform like Nuxeo.

Thomas found a site that references best practices on Javascript: http://superherojs.com/.

NB: As you will see, with JS, even coding rules can become hype!

Test drive and Connect

In the context of Next Generation Connect, Damien started to test some technology solutions:

• use CoffeeScript instead of vanilla Javascript
• use tools like YeoMan
  • Yo: template / scriptlet system
  • Bower simple dependencies management
  • Grunt: makefile
• do TDD in JS/Html5

Based on this first experience, we must define the blueprints of what the future Nuxeo JS Framework should be. So far we have mainly discussed the opportunity to use CoffeeScript vs plain JavaScript:

Pros:

• CoffeeScript is prettier and shorter than JavaScript
• CoffreeScript hides some of the quirk of Javascript
• the generate JS is clean
• we can maintain a bijection between JS <=> CoffeeScript
• there seem to be some traction on CoffeeScript

Cons:

• debugging needs to be done in pure JS (although a CoffeeScript source map now seems to be available)
• CoffeeScript does not solve / hide all the JavaScript quirks and pitfalls
• future of CoffeeScript is not clear whereas ECMAScript sustainability is good

We would really like to have your feedback on this. Please tell us what you think of Javascript in a Nuxeo Platform context.

The post [Nuxeo Tech Report] News from the Developer Front #5 appeared first on Nuxeo Blogs.