Today I’m digging out an old question about XMP support in Nuxeo. I’m actually going to widen this a bit and talk about metadata in Nuxeo. A limitation we have these days is the fact that you can only read metadata from a file, not write them. So let’s think about what this means.

A few questions first: do you write back metadata to files when you edit them in Nuxeo? Do you sync everything you can extract from the file? How do you map this to existing document properties? When should you do all of this?

What we need to know is which metadata from which type of file should we extract and/or write back to the file?

So we can define a specific mapping between document properties and file metadata. It will be differentiated by the file mime type, the type of the document and the file’s xPath. If we don’t specify any xPath, we can still use the BlobHolder to retrieve the mail file of a document.

Another requirement we can add is to specify different mapper class. Maybe you want to extract data using Tika, maybe you want to use ExifTool, or maybe you want to use proprietary software to extract some exotic metadata.

We also need to know where to store metadata. The use case for this is quite simple. Let’s take the File document type. We know it will have different kind of files attached to it. Some will have XMP metadata, some will have EXIF, some ITPC, etc. We have to store that metadata into schemas, but we don’t want to add all of them to a specific document type. Facets are very useful for this, as we can associate a schema to a facet. So we can specify in the mapping contribution which facets are required. This way we’ll be able to add them before doing the metadata mapping.

A sample contribution would look like this:

  <extension point="mapper" target="org.nuxeo.metadata.FileMetadataService">
    <mapper id="defaultTikaMapper" class="org.nuxeo.metadata.TikaDefaultMapper" />
  </extension>
  <extension point="mapping" target="org.nuxeo.metadata.FileMetadataService">
    <mapping id="tikaXMP" mapper="defaultTikaMapper" nxpath="files:files/item[0]/file">
      <mimeTypes>
        <mimeType>image/vnd.adobe.photoshop</mimeType>
        <mimeType>application/vnd.adobe.photoshop</mimeType>
        <mimeType>application/pdf</mimeType>
      </mimeTypes>
      <requirements>
        <facet>XMP</facet>
      </requirements>
      <properties>
        <propertyItem xpath="xmp:BitsPerSample" metadataName="tiff:BitsPerSample" policy="readonly" />
        <propertyItem xpath="xmp:ImageWidth" metadataName="tiff:ImageWidth" policy="readonly" />
        <propertyItem xpath="xmp:ImageLength" metadataName="tiff:ImageLength" policy="readonly" />
        <propertyItem xpath="xmp:CreatorTool" metadataName="xmp:CreatorTool" policy="readonly" />
        <propertyItem xpath="xmp:NPages" metadataName="xmpTPg:NPages" policy="readonly" />
      </properties>
    </mapping>
  </extension>
  <extension point="docMapping" target="org.nuxeo.metadata.FileMetadataService">
    <doc docType="File">
      <mappingId>tikaXMP</mappingId>
    </doc>
  </extension>

Now each time you want to create a new extension point, you need two things: the XML mapping file and the service managing the XP registration and unregistration.

If you have Nuxeo IDE, there’s a wizard that generates parts of the code. It’s called Nuxeo Component. You should have an XML file and a Java class.

The Java class comes with some useful comments about service implementation:

public class FileMetadataServiceImpl extends DefaultComponent {

    protected Bundle bundle;

    public Bundle getBundle() {
        return bundle;
    }

    /**
     * Component activated notification.
     * Called when the component is activated. All component dependencies are resolved at that moment.
     * Use this method to initialize the component.
     * <p>
     * The default implementation of this method is storing the Bundle owning that component in a class field.
     * You can use the bundle object to lookup for bundle resources:
     * <code>URL url = bundle.getEntry("META-INF/some.resource");</code>, load classes or to interact with OSGi framework.
     * <p>
     * Note that you must always use the Bundle to lookup for resources in the bundle. Do not use the classloader for this.
     * @param context the component context. Use it to get the current bundle context
     */
    @Override
    public void activate(ComponentContext context) {
        this.bundle = context.getRuntimeContext().getBundle();
    }

    /**
     * Component deactivated notification.
     * Called before a component is unregistered.
     * Use this method to do cleanup if any and free any resources held by the component.
     *
     * @param context the component context. Use it to get the current bundle context
     */
    @Override
    public void deactivate(ComponentContext context) {
        this.bundle = null;
    }

    /**
     * Application started notification.
     * Called after the application started.
     * You can do here any initialization that requires a working application
     * (all resolved bundles and components are active at that moment)
     *
     * @param context the component context. Use it to get the current bundle context
     * @throws Exception
     */
    @Override
    public void applicationStarted(ComponentContext context) throws Exception {
        // do nothing by default. You can remove this method if not used.
    }

}

<component name="org.nuxeo.metadata.FileMetadataService" version="1.0">
  <implementation class="org.nuxeo.metadata.FileMetadataServiceImpl" />
</component>

The implementation tag points to the implementation of our service. We re going to add the provide tag that will point to the interface of our service managing the XP. Let’s call this interface FileMetadataService and leave it empty for the moment.

<component name="org.nuxeo.metadata.FileMetadataService" version="1.0">
  <service>
    <provide
      interface="org.nuxeo.metadata.FileMetadataService" />
  </service>
  <implementation class="org.nuxeo.metadata.FileMetadataServiceImpl" />
</component>

Now let’s write a test for that. Our goal is to make sure our service is working correctly.

package org.nuxeo.metadata.test;

import static org.junit.Assert.assertNotNull;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.nuxeo.ecm.core.test.CoreFeature;
import org.nuxeo.metadata.FileMetadataService;
import org.nuxeo.runtime.api.Framework;
import org.nuxeo.runtime.test.runner.Deploy;
import org.nuxeo.runtime.test.runner.Features;
import org.nuxeo.runtime.test.runner.FeaturesRunner;

@RunWith(FeaturesRunner.class)
@Features(CoreFeature.class)
@Deploy({ "nuxeo-platform-filemanager-metadata" })
public class FileMetadataServiceTest {

    @Test
    public void testService() throws Exception {
        FileMetadataService serviceInterface = Framework.getService(FileMetadataService.class);
        assertNotNull(serviceInterface);
    }

}

Our test is retrieving the service using the interface name correctly. We can now move to the extension point implementation. Following the first example, We’re going to have three different points. One for the mapper class, one for the actual mappings and one to associate a document type to a mapping. As they are all different, we’re going to need at least three different XMAP descriptor.

Here’s the XML declaring the service and the three new extension points:

<?xml version="1.0"?>
<component name="org.nuxeo.metadata.FileMetadataService" version="1.0">

  <service>
    <provide
      interface="org.nuxeo.metadata.FileMetadataService" />
  </service>

  <implementation class="org.nuxeo.metadata.FileMetadataServiceImpl" />

  <extension-point name="mapper">
    <object
      class="org.nuxeo.metadata.MetadataMapperDescriptor" />
  </extension-point>

  <extension-point name="mapping">

    <object
      class="org.nuxeo.metadata.MetadataMappingDescriptor" />
  </extension-point>

  <extension-point name="docMapping">
    <object
      class="org.nuxeo.metadata.DocMetadataMappingDescriptor" />
  </extension-point>

</component>

Here’s the mapper descriptor. It’s really simple as we just need a class and a name in case someone would want to overwrite it.

package org.nuxeo.metadata;

import org.nuxeo.common.xmap.annotation.XNode;
import org.nuxeo.common.xmap.annotation.XObject;

@XObject("mapper")
public class MetadataMapperDescriptor {

    @XNode("@id")
    protected String id;

    @XNode("@class")
    private Class<MetadataMapper> adapterClass;

    public String getId() {
        return id;
    }

    public MetadataMapper getMapper() throws InstantiationException,
            IllegalAccessException {
        return adapterClass.newInstance();
    }

}

Here’s the mapping descriptor. That’s the complicated one :)

package org.nuxeo.metadata;

import org.nuxeo.common.xmap.annotation.XNode;
import org.nuxeo.common.xmap.annotation.XNodeList;
import org.nuxeo.common.xmap.annotation.XObject;

@XObject("mapping")
public class MetadataMappingDescriptor {

    @XNode("@id")
    protected String id;

    @XNode("@nxpath")
    protected String nxpath;

    @XNode("@mapper")
    protected String mapperId;

    @XNodeList(value = "mimeTypes/mimeType", componentType = String.class, type = String[].class)
    protected String[] mimeTypes;

    @XNodeList(value = "requirements/schema", componentType = String.class, type = String[].class)
    protected String[] requiredSchema;

    @XNodeList(value = "requirements/facet", componentType = String.class, type = String[].class)
    protected String[] requiredFacets;

    @XNodeList(value = "properties/propertyItem", componentType = PropertyItemDescriptor.class, type = PropertyItemDescriptor[].class )
    protected PropertyItemDescriptor[] properties;

    public String getId() {
        return id;
    }

    public String getNxpath() {
        return nxpath;
    }

    public String getMapperId() {
        return mapperId;
    }

    public String[] getMimeTypes() {
        return mimeTypes;
    }

    public String[] getRequiredSchema() {
        return requiredSchema;
    }

    public String[] getRequiredFacets() {
        return requiredFacets;
    }

    public PropertyItemDescriptor[] getProperties() {
        return properties;
    }

}

And finally the document type to mapping descriptor:

package org.nuxeo.metadata;

import org.nuxeo.common.xmap.annotation.XNode;
import org.nuxeo.common.xmap.annotation.XNodeList;
import org.nuxeo.common.xmap.annotation.XObject;

@XObject("doc")
public class DocMetadataMappingDescriptor {

    @XNode("@docType")
    protected String docType;

    @XNodeList(value = "mappingId", type = String[].class, componentType = String.class)
    protected String[] mappingId;

    @XNodeList(value = "mapping", type = MetadataMappingDescriptor[].class, componentType = MetadataMappingDescriptor.class)
    protected MetadataMappingDescriptor[] innerMapping;

    public String getDocType() {
        return docType;
    }

    public String[] getMappingId() {
        return mappingId;
    }

    public MetadataMappingDescriptor[] getInnerMapping() {
        return innerMapping;
    }

}

Now that we have all of this, we need to add some code in the service implementation to register the XML associated with thoses descriptor. Everything starts from the registerContribution method. Depending on the extensionPoint name, we cast the contribution as one of the descriptor and give to the appropriate method registerSomething method. The most interesting method here is registerDocMapping. It’s where we start to sotre nicely the mappings, so that they’ll be easy to retrieve for one particular DocumentModel.

package org.nuxeo.metadata;

import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import org.nuxeo.ecm.core.api.Blob;
import org.nuxeo.ecm.core.api.ClientException;
import org.nuxeo.ecm.core.api.CoreSession;
import org.nuxeo.ecm.core.api.DocumentModel;
import org.nuxeo.ecm.core.api.blobholder.BlobHolder;
import org.nuxeo.runtime.model.ComponentContext;
import org.nuxeo.runtime.model.ComponentInstance;
import org.nuxeo.runtime.model.DefaultComponent;

/**
 * @author ldoguin
 * @since 5.7
 */
public class FileMetadataServiceImpl extends DefaultComponent implements
        FileMetadataService {

    /**
     * List of Mappers
     */
    protected Map<String, MetadataMapper> mappers;

    /**
     * Every mapping
     */
    protected Map<String, MetadataMappingDescriptor> mappings;

    /**
     * Every mappings by mimeType
     */
    protected Map<String, MetadataMappingDescriptor> mimeTypeMappings;

    /**
     * Every mappings withouth specific nxpath, use blob holder instead
     */
    protected Map<String, MetadataMappingDescriptor> bhMapping;

    /**
     * Three level mapping registry.
     * First key is document type
     * Second key is nxPath to the blob
     * Third key is mime type
     */
    protected Map<String, Map<String, Map<String, MetadataMappingDescriptor>>> nxPathMapping;

    @Override
    public void activate(ComponentContext context) throws Exception {
        mappings = new HashMap<String, MetadataMappingDescriptor>();
        mimeTypeMappings = new HashMap<String, MetadataMappingDescriptor>();
        mappers = new HashMap<String, MetadataMapper>();
        bhMapping = new HashMap<String, MetadataMappingDescriptor>();
        nxPathMapping = new HashMap<String, Map<String, Map<String, MetadataMappingDescriptor>>>();
    }

    @Override
    public void registerContribution(Object contribution,
            String extensionPoint, ComponentInstance contributor)
            throws Exception {
        if (extensionPoint.equals("mapper")) {
            if (contribution instanceof MetadataMapperDescriptor) {
                registerMapper((MetadataMapperDescriptor) contribution);
            }
        } else if (extensionPoint.equals("mapping")) {
            if (contribution instanceof MetadataMappingDescriptor) {
                registerMapping((MetadataMappingDescriptor) contribution);
            }
        } else if (extensionPoint.equals("docMapping")) {
            if (contribution instanceof DocMetadataMappingDescriptor) {
                registerDocMapping((DocMetadataMappingDescriptor) contribution);
            }
        }
    }

    private void registerMapping(MetadataMappingDescriptor contribution) {
        mappings.put(contribution.getId(), contribution);
        String[] mimetypes = contribution.getMimeTypes();
        for (String mimeType : mimetypes) {
            mimeTypeMappings.put(mimeType, contribution);
        }
    }

    private void registerDocMapping(DocMetadataMappingDescriptor contribution) {
        String docType = contribution.docType;
        MetadataMappingDescriptor[] innerMappings = contribution.getInnerMapping();
        for (MetadataMappingDescriptor metadataMappingDescriptor : innerMappings) {
            addMappingToRegistries(docType, metadataMappingDescriptor);
        }
        String[] mappingIds = contribution.getMappingId();
        for (String string : mappingIds) {
            addMappingToRegistries(docType, mappings.get(string));
        }
    }

    private void addMappingToRegistries(String docType,
            MetadataMappingDescriptor metadataMappingDescriptor) {
        String nxPath = metadataMappingDescriptor.getNxpath();
        if (nxPath != null && !"".equals(nxPath)) {
            Map<String, Map<String, MetadataMappingDescriptor>> docNxPathMapping = nxPathMapping.get(docType);
            if (docNxPathMapping == null) {
                docNxPathMapping = new HashMap<String, Map<String, MetadataMappingDescriptor>>();
            }
            Map<String, MetadataMappingDescriptor> mimeTypeToMapper = docNxPathMapping.get(nxPath);
            if (mimeTypeToMapper == null) {
                mimeTypeToMapper = new HashMap<String, MetadataMappingDescriptor>();
                docNxPathMapping.put(nxPath, mimeTypeToMapper);
            }
            String[] mimeTypes = metadataMappingDescriptor.getMimeTypes();
            for (String mimeType : mimeTypes) {
                mimeTypeToMapper.put(mimeType, metadataMappingDescriptor);
            }
            nxPathMapping.put(docType, docNxPathMapping);
        } else {
            String[] mimeTypes = metadataMappingDescriptor.getMimeTypes();
            for (String mimeType : mimeTypes) {
                String id = docType + mimeType;
                bhMapping.put(id, metadataMappingDescriptor);
            }
        }
    }

    private void registerMapper(MetadataMapperDescriptor contribution)
            throws InstantiationException, IllegalAccessException {
        mappers.put(contribution.id, contribution.getMapper());
    }

    @Override
    public List<MetadataMappingDescriptor> getMappings(DocumentModel doc)
            throws ClientException {
        String docType = doc.getType();
        List<MetadataMappingDescriptor> mappings = new ArrayList<MetadataMappingDescriptor>();
        BlobHolder bh = doc.getAdapter(BlobHolder.class);
        if (bh != null) {
            Blob blob = bh.getBlob();
            if (blob != null) {
                String blobMimeType = blob.getMimeType();
                String bhId = docType + blobMimeType;
                MetadataMappingDescriptor bhMapper = bhMapping.get(bhId);
                if (bhMapper != null) {
                    mappings.add(bhMapper);
                }
            }
        }
        Map<String, Map<String, MetadataMappingDescriptor>> nxPathDocMapping = nxPathMapping.get(docType);
        if (nxPathDocMapping != null) {
            for (String nxPath : nxPathDocMapping.keySet()) {
                Blob blob = (Blob) doc.getPropertyValue(nxPath);
                if (blob != null) {
                    String blobMimeType = blob.getMimeType();
                    MetadataMappingDescriptor bhMapper = nxPathDocMapping.get(
                            nxPath).get(blobMimeType);
                    if (bhMapper != null) {
                        mappings.add(bhMapper);
                    }
                }
            }
        }
        return mappings;
    }

}

Let’s also add a method called getMappings to our interface and implement it. It will retrieve the list of mapping descriptor for a specific document. That’s what we’re going to use in our previous unit test to make sure our extension points are working nicely. Let’s add some mapping examples for our test:

<component name="org.nuxeo.metadata.test.contrib">

  <extension target="org.nuxeo.ecm.core.schema.TypeService"
             point="doctype">
    <doctype name="File2" extends="File">
    </doctype>
  </extension>

  <extension point="mapper" target="org.nuxeo.metadata.FileMetadataService">
    <mapper id="defaultTikaMapper" class="org.nuxeo.metadata.TikaDefaultMapper" />
    <mapper id="testTikaMapper" class="org.nuxeo.metadata.test.TestTikaDefaultMapper" />
  </extension>

  <extension point="mapping" target="org.nuxeo.metadata.FileMetadataService">
    <mapping id="tikaPDF" mapper="defaultTikaMapper">
      <mimeTypes>
        <mimeType>application/pdf</mimeType>
        <mimeType>application/x-pdf</mimeType>
      </mimeTypes>
      <requirements>
        <schema>dublincore</schema>
        <facet>xmp</facet>
      </requirements>
      <properties>
        <propertyItem xpath="xmp:pagecount" metadataName="pagecount" policy="readonly" />
        <propertyItem xpath="dc:title" metadataName="title" policy="sync" />
      </properties>
    </mapping>
    <mapping id="tikaVideo" mimeType="video/mpeg" mapper="defaultTikaMapper">
      <mimeTypes>
        <mimeType>video/quicktime</mimeType>
        <mimeType>video/mp4</mimeType>
        <mimeType>video/mpeg</mimeType>
      </mimeTypes>
      <requirements>
        <schema>dublincore</schema>
        <facet>xmp</facet>
      </requirements>
      <properties>
        <propertyItem xpath="dc:title" metadataName="title" policy="sync" />
      </properties>
    </mapping>
  </extension>

  <extension point="docMapping" target="org.nuxeo.metadata.FileMetadataService">

    <doc docType="File2">
      <mapping nxpath="files:files/item[0]/file" mapper="defaultTikaMapper">
        <mimeTypes>
          <mimeType>image/png</mimeType>
        </mimeTypes>
        <requirements>
          <schema>dublincore</schema>
          <facet>xmp</facet>
        </requirements>
        <properties>
          <propertyItem xpath="dc:title" metadataName="title" policy="sync" />
        </properties>
      </mapping>
  </doc>

  <doc docType="File">
      <mappingId>tikaPDF</mappingId>
      <mappingId>tikaVideo</mappingId>
  </doc>
  </extension>

</component>

Now we can add some code to test those mappings:

package org.nuxeo.metadata.test;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotNull;

import java.io.File;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.nuxeo.common.utils.FileUtils;
import org.nuxeo.ecm.core.api.Blob;
import org.nuxeo.ecm.core.api.CoreSession;
import org.nuxeo.ecm.core.api.DocumentModel;
import org.nuxeo.ecm.core.api.impl.blob.FileBlob;
import org.nuxeo.ecm.core.test.CoreFeature;
import org.nuxeo.metadata.FileMetadataService;
import org.nuxeo.metadata.FileMetadataServiceImpl;
import org.nuxeo.metadata.MetadataMapper;
import org.nuxeo.metadata.MetadataMappingDescriptor;
import org.nuxeo.metadata.PropertyItemDescriptor;
import org.nuxeo.runtime.api.Framework;
import org.nuxeo.runtime.test.runner.Deploy;
import org.nuxeo.runtime.test.runner.Features;
import org.nuxeo.runtime.test.runner.FeaturesRunner;
import org.nuxeo.runtime.test.runner.LocalDeploy;

import com.google.inject.Inject;

@RunWith(FeaturesRunner.class)
@Features(CoreFeature.class)
@Deploy({ "nuxeo-platform-filemanager-metadata",
        "nuxeo-platform-filemanager-metadata-test" })
@LocalDeploy({
        "nuxeo-platform-filemanager-metadata-test:OSGI-INF/metadata-core-contrib.xml",
        "nuxeo-platform-filemanager-metadata-test:OSGI-INF/test-tika-contrib.xml" })
public class FileMetadataServiceTest {

    @Inject
    CoreSession session;

    @Test
    public void testService() throws Exception {
        FileMetadataService serviceInterface = Framework.getService(FileMetadataService.class);
        assertNotNull(serviceInterface);
        DocumentModel file = createFileDocumentModelWithPdf();
        List<MetadataMappingDescriptor> fileMappings = serviceInterface.getMappings(file);
        assertNotNull(fileMappings);
        assertFalse(fileMappings.isEmpty());
        assertEquals(1, fileMappings.size());

        DocumentModel file2 = createFileDocumentModelWithPng();
        fileMappings = serviceInterface.getMappings(file2);
        assertNotNull(fileMappings);
        assertFalse(fileMappings.isEmpty());
        assertEquals(1, fileMappings.size());

        MetadataMappingDescriptor mapping = fileMappings.get(0);
        String[] facets = mapping.getRequiredFacets();
        assertNotNull(facets);
        assertEquals(1, facets.length);
        assertEquals("xmp", facets[0]);
        String[] schemas = mapping.getRequiredSchema();
        assertNotNull(schemas);
        assertEquals(1, schemas.length);
        assertEquals("dublincore", schemas[0]);

        assertEquals("image/png", mapping.getMimeTypes()[0]);
        assertEquals("files:files/item[0]/file", mapping.getNxpath());
        assertEquals("defaultTikaMapper", mapping.getMapperId());
        PropertyItemDescriptor[] properties = mapping.getProperties();
        assertNotNull(properties);
        assertEquals(1, properties.length);

        PropertyItemDescriptor item = properties[0];
        assertNotNull(item);
        assertEquals("title", item.getMetadataName());
        assertEquals("dc:title", item.getXpath());
        assertEquals("sync", item.getPolicy());

    }

    private DocumentModel createFileDocumentModelWithPdf() throws Exception {
        DocumentModel doc = session.createDocumentModel("/", "file", "File");
        File f = FileUtils.getResourceFileFromContext("data/hello.pdf");
        Blob blob = new FileBlob(f);
        blob.setMimeType("application/pdf");
        doc.setPropertyValue("file:content", (Serializable) blob);
        return doc;
    }

    private DocumentModel createFileDocumentModelWithPng() throws Exception {
        DocumentModel doc = session.createDocumentModel("/", "file2", "File2");
        File f = FileUtils.getResourceFileFromContext("data/training.png");
        Blob blob = new FileBlob(f);
        blob.setMimeType("image/png");
        Map<String, Serializable> blobMap = new HashMap<String, Serializable>();
        blobMap.put("file", (Serializable) blob);
        blobMap.put("filename", "training.png");
        List<Map<String, Serializable>> blobs = new ArrayList<Map<String, Serializable>>();
        blobs.add(blobMap);
        doc.setPropertyValue("files:files", (Serializable) blobs);
        return doc;
    }

}

That’s it for today. Next time I’ll show you how to actually do the mapping :)

The post [Monday Dev Heaven] Extract Metadata from Content File Attachments appeared first on Nuxeo Blog.

I am sure you always dreamed about this, but didn’t dare to ask: a User Interface Style Guide. We recently created this tool, focused on the Nuxeo Platform. As the Nuxeo Platform is designed for customizability, the Nuxeo UI Style Guide will help you learn how to design the interface to fit your end users’ needs.

What’s in this guide?

• It explores all the available CSS rules from the default platform,
• It gives you tips and tricks to design a page layout with a grid,
• It helps you design the user interaction you need with actions and feedback,
• It lists all the available icons in your running platform.

What will you gain with it?
You will gain in consistency. Following the UI design patterns, the platform will increase its value by being clear and predictable, and so adoptable easily and quickly by users.
You will improve the usability. By being consistent in your interface, users will rapidly feel confortable and trust your platform.
You will gain in maintainability. As styles are there to evolve depending on your needs, the visual explorer is a useful tool to have an overview of the existings rules you can use in Nuxeo Studio, for example.
You will gain in compatibility. These rules and recommendations stand as a model to follow, as a guarantee that we are all evolving in the same direction to build our platform.

In detail, here is the UI Style Guide Table of Contents:
* Recommendations: the best practices to follow before starting a customization project
* Grid Layout: explains how to use the grid to design your tabs
* Tables: shows you the different table styles for your data tables and forms
* Messages: a list of all the types of feedback you can give to your user
* Titles: helps you to build your information architecture
* Dialogs: shows which modal pop-ups are running in the platform
* Buttons: lists the different kinds of actions and how you can style them
* Navigation: displays the blocks and buttons to browse your platform
* Boxes: compiles the default box models available
* Helpers: stands for a list of CSS classes to tweak a default behavior
* Icons: lists by category all the icons in your current platform.

This guide is here to be enriched over time, and follow the platform releases.

So now developers, integrators and all inquiring minds, discover and pick all the CSS rules you want to use in your project, follow the recommendations and the design patterns, play and combine styles for your new projects, and send us your feedback.

Tell us what you think, how it is useful to your project, what is missing, and which other tools you dreamed about to complete your Nuxeo development environment by using the comments or sending us an email to feedback@nuxeo.com.

Thank you.

The post The Nuxeo UI Style Guide appeared first on Nuxeo Blog.

Last week I attended the final review meeting for the SAMAR project (website in French). SAMAR (Station d’Analyse Multimedia en langue ARabe) is a multi-enterprise project with the objective of developing a platform to manage multimedia content in Arabic.

The goal of the project was to develop tools for the automated analysis of Arabic news content (text, audio and video) from Agence France Presse for smart multilingual searching with (among other things):

• speech to text transcription from the audio track of videos
• semantic text analysis and linking for arabic text, with categorization by IPTC topics
• translation to French and English.

Today, the processing platforms for the Arabic language still lack maturity. Semantic processing in Arabic is particularly difficult. It’s not because of the nature of the character themselves, since Arabic uses a 24-letter alphabet and word composition that is not very different from Latin languages. It’s more about the fact that vocals are often not written, so they have to be deduced from the context by the algorithm. Transliterations from English names into Arabic often have a lot of variability, and Arabic has many dialects that share many language features but are still quite different (both from a pronunciation standpoint and a vocabulary standpoint).

Also, the relative lack of linguistically annotated corpus of documents with respect to English (for instance) makes it more challenging and costly to build good models for speech transcription, translation and semantic analysis (topic categorization and named entity detection).

With the SAMAR project, we have developed a working platform for semantic processing of Arabic language multimedia content. The project work will be validated by conducting experiments on all Arabic dispatches already produced by the AFP (about one million dispatches, representing more than 150 million words), as well as radio and television data in Arabic.

A demo is worth a thousand words, so view the video to see the proof of concept in action!

The post Semantic Processing of Multimedia Content in Arabic appeared first on Nuxeo Blog.

Recently, a prospective Nuxeo client, while evaluating document management software, asked us to demonstrate Nuxeo system performance and scalability capabilities in a proof-of-concept experiment. His application deals with a massive number of documents, and requires rapid response times for a large number of simultaneous users. The goal of this proof-of-concept experiment was to see if Nuxeo was up to the task.

We were happy to jump into the ring–our engineers are keen sports for these sorts of activities. Now that the dust has settled, and the benchmarks are finished, we thought we would share the results. We have not taken the time to provide much additional information on the performance of the Nuxeo Platform, so this provides us with that opportunity. The results are in line with our other benchmarks, and more than ever confirm that Nuxeo is best-in-class when it comes to performance and scalability!

The Set Up

One of the goals of the experiment was to assess the performance of the system running on a complex architecture. This was not a simple Nuxeo installation, as is the case for most of our continuous performance testing, but rather a cluster setup. Our team decided to run the benchmarks on an architecture made of two nuxeo application servers and four repository servers. We also decided to run this architecture on a standard public cloud infrastructure, which we know is not optimum for response times and scalability. In any event, high-end, on-premises hardware would give better numbers, but would also require a longer set up, and we only had a couple days for this project.

Here, we have to mention that Cloud Computing technology has changed the way we execute this kind of project. We make the simple, efficient deployment of the Nuxeo Platform a priority, especially for our Nuxeo Cloud content management offering. Thanks to that, we had the knowledge and the tools to quickly deploy a complex architecture on the Cloud. The Cloud we used here (as well as for our Nuxeo Cloud offering) is Amazon Web Services (AWS), but of course this could be set up with other cloud providers.

We ended up with two Nuxeo servers running in cluster mode behind elastic load balancing, connected to four content repositories, each running on a dedicated PostgreSQL database. We stored the binary data on Amazon’s S3 blob service — a built-in feature for Nuxeo.

The Test

The first challenge of a test like this is to load the system with enough data. In our case, we weighed our Nuxeo installation down with 66 million documents. Each document consisted of a simple document object in the Nuxeo repository, with basic metadata, and an image that, for instance, would be the result of a capture process on a remote system, e.g. account statements, invoices, or other business documents.

The test scenario was a snap. The test requirements were actually a fairly straightforward use of the Nuxeo Document Management module. Concretely, the test consisted of a sequence of searches in the document repository, displaying the document, and downloading binary files attached in the browser. All this was to be done via the standard Nuxeo Document Management user interface, which is a JSF application. Here again, there is undoubtedly room for improvement. Running the same benchmark directly through the Nuxeo API would also be interesting, and by “interesting” we mean faster!

Performance aside, what we were really interested in assessing was stability. With that in mind, we took the user sequence described above and simply executed it for a multitude of simultaneous users over a long period; this activity on the application was generated by Funkload, our load testing tool of choice.

Funkload is home-brew software that has been developed over the years with Benoit Delbosc as the lead developer of the project. Like much of what Nuxeo does, Funkload has been developed in an open source manner. The software is open source and we’ve been inviting other people to contribute and use it as much as we do. Today, seven years after the first version, Funkload is a remarkable, high-quality piece of software with a very active community. While it has little to do directly with the content management solutions that Nuxeo brings to the market–which may be a reason why we don’t talk that much about Funkload–it is as much a part of the Nuxeo development infrastructure as other more well-known components. If at some point you need load testing software, whether to stress a Nuxeo application, or some other platform, we highly recommend Funkload for serious stress tests!

Results and Perks

Here is what the test told us: the platform consistently delivers 70 pages of content per second with 150 concurrent users harnessing the system, making requests every second, and downloading content. The platform is very stable on test runs of up to 5 hours. According to the Funkload reporting, the main limiting factor appears to simply be the I/O of the system–the hard drives.

In our opinion, 70 pages per second of full content access for 150 concurrent users on a complex cluster system, and with a repository loaded with 66 million documents is already a great outcome. Then, take into account that 150 concurrent users harnessing the platform every second is much higher than the typical usage of an ECM or document management platform. We can safely extrapolate this number to something like 1500 concurrent users in a real-world production environment.

No other content management system can replicate these kinds of results. Add to this the fact that it was done on virtual, cloud-based commodity servers, and you have the first takeaway:

Nuxeo and its Cloud capabilities are a major step forward, making Cloud Document Management a reality for an enterprise-grade project. Together with AWS or a comparable IaaS service, it is now possible to set up a dedicated complex architecture that delivers high-performing, highly scalable content and a document management solution for a fraction of the cost of yesterday’s solutions. The fact that it is cloud-based may take a backseat to the real story: this level of performance for the price constitutes a game-changer. Cloud computing is ready to run high-performance, heavy-load and content-centric applications, at least if they are running the Nuxeo Platform.

The second conclusion we draw from this benchmark is that these limits can be pushed much further. The architecture set up in question is totally scalable horizontally. Imagine for a moment that you want to add repositories or nodes to the Nuxeo cluster. Not a problem–the numbers increase linearly.

What about scaling vertically? Can we improve the intrinsic performance of the system? If we swap out our Medium AWS machines for better ones, we’ll see an improvement. Or, for that matter, let’s imagine we go even further, and move to dedicated hardware. We’ll examine that I/O limitation we encountered in our Funkload testing, and we decide to tune a high-end file storage back-end system. Imagine we fine tune the other components in the setup. What numbers would we get then? Twice as fast? Two-thirds? Honestly, we can’t say, but what matters is that we know there is a lot of room for improvement for critical large scale applications, and there are two aspects than can be leveraged: horizontal and vertical scalability.

This post is just a short summary of the benchmark test. We’ve been running other benchmarks as well, and we constantly monitor the performance of our platform to make sure it evolves in the right direction. Please contact us if you are interested in this topic, and want more technical insight.

The post Nuxeo Platform Performance and Scalability: Beyond Passing the Test appeared first on Nuxeo Blog.

The Nuxeo UI Style Guide

I recently published a UI Style Guide for the Nuxeo Platform, to help developers and designers learn how to design your platform interface.

A first post was published to explain what’s inside and why you should follow it. In this post I will focus on the importance of setting up a guide like this, how developers are way more happy now, and how i learned a lot about my own design process.

A toolkit to make better use of everyone’s time
Developers were wasting a lot of time trying to start styling a bit their features, without having any rules to follow. It was not satisfying for them, and even less for me. The thing is, developers are proud of their feature once they see it cleaned and styled.
In my designer side, I was also wasting a lot of time reviewing their code, updating the layout, adding missing elementary classes, and trying to fight to keep the screens consistent.

Clearly, the developers would be even prouder of their work if they had the toolkit to clean and style their features on their own.

The solution was obvious: they needed a UI Style Guide to rely on stable rules: a guide with all the available styles, icons, and recommendations to keep the platform simple, clear, and usable.

A helpful process to improve your design rules
As exciting as it might be to draft a guide, dreaming and thinking about how clear and simple it will be for everyone once finished, when I started I faced a list of challenges.

The first one was cleaning all the existing styles rules. I started a thorough investigation of existing elements, remove all the duplicated, old or unused rules, I made the CSS files easier to read, and I’ve made the comments and documentation more helpful in the files. This was a really satisfying step.

Then, I faced another thing: the styles are clean, great. But they aren’t complete. A key point was missing: my CSS rules were not flexible enough. I designed a box, but I didn’t provide a way to add a button in it. And what if I need a title and a button in the same line? I started to address the list of all the common design patterns developers and users need, and implement them in my CSS rules. Global design patterns are more useful than specific design, so I rewrote a lot of rules as generic styles. This was a nice way to remodel the patterns and make them robust.

Also, by adding an icon explorer, even if I knew that some icons were sometimes duplicated in the platform, having the list in front of my eyes is the best tool I could provide for myself. With that, I could forget about an unsuccessful grep search command: I only have to scroll to see which icons I have to update. This provides a visual tool to check the visual consistency of your icons.

At last, once the code and rules were cleaned and well explained in the CSS files, I started to write the guide. This was the biggest challenge:  justifying my design decisions by explaining the layout and class patterns. If you’re not able to explain or justify plainly a structure or a pattern, something is wrong. It was the most interesting task as a designer, as I had to understand which errors I made and why, and which solution is much simpler to implement. This was a great process to institute a sustainable guide for the design.

What I’ve gained, what I learned

• I now have an easily readable coding style and a set of clean CSS rules that is easy to maintain.
• I stopped designing the same element each month: the Style Guide is here to show how to design one rollover menu for good
•  I stopped designing each element as it is the only page existing in the platform. The interface is a flow, not a list of independent steps. I now design elements thinking of this flow.
• Toolkits save a lot of time. Developers have more time to focus on their features, and designers have more time to focus on design pattern quality.
• Sharing is important in the design process. I can directly see with them what is missing. They can directly tell me what is unclear.
• Using available styles as a standard, I have now a solid and stable base to build on.

The importance of trusting your design choices
With a guide like this, everyone will understand that each pattern is done for a reason. This will result in a better user experience through consistency and homogenization, and all your patterns will be applied. And you will see if it works: after all the lists of all the advantages a guide can bring to designer, this last thing is the most important.

By trusting your design decisions and following them, you will get feedback from others. This is the only way to be sure they work. It is difficult to be sure a design pattern will be widely adopted, so trust your choices and measure their efficiency. This is the best way to see what worked and what didn’t, so you can always continue to improve.

Don’t be disappointed if something is not working. It is not a failure. It is the best occasion to keep improving.

The post Why You Need a UI Style Guide appeared first on Nuxeo Blog.

Nuxeo Studio 2.9 has been released. If you are a customer, you already have access to this new version, because it’s a hosted environment. You just get the gift of new features and improvements. Happy Holidays!

Highlights of this version include:

• Multi-user mode: allows multiple users to collaborate on a Nuxeo Studio project simultaneously, improving productivity. An auto-locking mechanism with real time releasing and presence management was implemented to support this feature. If you’re already a Nuxeo Connect client, and you’d like to have access to this feature, please contact your account manager or the support team.
• Improved complex type management with the ability to configure content views that leverage complex queries and to display complex properties in the result set, without limiting the depth of the complex schema, and without performance costs.
• Improved workflow graph: the generic square has been replaced by more explicit symbols, to better indicate start and merge/fork nodes on the workflow graph.
• Improved form management in the tab designer, as well as better control of component display.
• Faceted search configuration: enables definition of multiple named sets of filters on the faceted search module. This feature is available on the 5.7 version of the Nuxeo Platform, which is currently in development. You are welcome to download and install the 5.7 version so you can play with this new feature.

For full details on Nuxeo Studio 2.9, see the release notes.

This short screencast demonstrates how easy it is to configure faceted search with Nuxeo Studio.

If you’re not already using Nuxeo Studio, you should be. Nuxeo Studio helps application developers and solution architects speed time to deployment of their content management applications – from weeks to days – with a graphical customization and configuration toolset, reducing or eliminating the need for custom code. To start a 30-day trial of Nuxeo Studio, simply download the Nuxeo Platform and follow the installation wizard.

The post Nuxeo Studio 2.9 – What’s New? appeared first on Nuxeo Blog.

Here’s a question from NDeveloper. He wants to know how to get the parent ID of a vocabulary entry using its label. A couple of words about vocabularies first.

A vocabulary is a list of labels that are used in the application to populate the various selection lists (drop-down, select or multi-select lists). A vocabulary can be used independently or it can be associated with other vocabularies to compose a multi-level list.

You could add one easily with Nuxeo Studio, just take a look at the documentation.

Now here’s a more technical definition:

A vocabulary is a specialized directory with only a few important columns that are used by Nuxeo Platform to display things like menus and selection lists.

In Nuxeo Platform, a directory is a source of (mostly) table-like data that lives outside of the VCS document storage database. A directory is typically a connection to an external data source that is also access by other processes than Nuxeo itself (therefore allowing shared management and usage).

So to answer that question,  we’re going to use the directory API:

import java.io.Serializable;
import java.util.HashMap;
import java.util.Map;

import org.nuxeo.ecm.core.api.DocumentModel;
import org.nuxeo.ecm.core.api.DocumentModelList;
import org.nuxeo.ecm.directory.Session;
import org.nuxeo.ecm.directory.api.DirectoryService;
import org.nuxeo.runtime.api.Framework;

public class VocabularyExample {

   public void fihdParentId() throws Exception {
        // Open the directory
        DirectoryService dirService = Framework.getService(DirectoryService.class);
        Session directorySession = dirService.open("nameOfMyVocabulary");
        // Create a query filter
        Map queryFilter = new HashMap();
        queryFilter.put("label", "childLabel");
        // Execute the query, wrapped in a DocumentModel list
        // We might get many entries as we're doing a query on the label, not
        // the id
        DocumentModelList queryResult = directorySession.query(queryFilter);
        // If you have a child vocabulary or a hierarchical one, it means the
        // schema used for your directory is xvocabulary which has 5 columns
        // named id, label, parent, obsolete, ordering. Let's assume the query
        // returned only one entry.
        DocumentModel entry = queryResult.get(0);
        String parentId = entry.getProperty("xvocabulary:parent").getValue(
                String.class);

        // It's of course much easier if you have the child id instead of the
        // label because you don't need a query.
        entry = directorySession.getEntry("childId");
        parentId = entry.getProperty("xvocabulary:parent").getValue(
                String.class);

        // Whatever you do, do not forget to close the directory session!
        directorySession.close();
    }
}

Don’t forget that it’s possible to have several entries with the same label. So it’s much better to use the child ID when possible.

The post [Q&A Friday] Get Parent ID from Child Label of a Nuxeo Vocabulary appeared first on Nuxeo Blog.

Hello everyone and happy holidays! I have a new Nuxeo Tech Report for you – taken straight from the meetings of the Nuxeo Dev Team.

Nuxeo Studio

Nuxeo Studio 2.9 has been released and we’ve added some interesting new features.

We now have locking and presence management:

• project displays connected user names
• when a user access a feature
  • it is automatically locked if possible
  • if the feature is already locked the lock owner name is displayed as well as a “steal lock”
  • when lock is released the waiting user will receive a notification and a link to reload
• heartbeat (no more session timeout!)

This has been built using Vert.X.

Widgets are now associated with a “control” that can define:

• if the widget should have it’s own form
• if widget manages it’s label
• if widget needs an ajax form

Which means:

• better management of summary widgets
• New widgets for actions (with support for FancyBox)
• Widget for tabs
• Content Routing forms

Nuxeo DAM

We’re working on a new DAM Asset Browser View that:

• is configurable from Nuxeo Studio
• shares as much code as possible code with Nuxeo Document Management :)

Thumbnail Adapter

The goal is to have an adapter that provides:
– a default thumb picture
– a series of named thumbs? Let us know what you think :)

Integration inside Nuxeo CAP includes:
– create a thumb widget
– add a Codec URL
– plug it in the default “big icons” result Layout

Nuxeo Drive Status

(because I know you’re curious about this!)
Antoine and Olivier are still working on Nuxeo Drive. The MP package and the client distributions have been completed and we now have a MacBook on the CI chain to build the MacOS package.

Talend Integration

An initial integration was started between Nuxeo and Talend. The goal is to have Talend components to read from Nuxeo and write to Nuxeo. See NXP-10615.

In the process we’ll work on Content Automation:
– create additional QueryAndFetch operation NXP-10614
– improve default marshaling
– add sample unit tests to manage complex types : you can see the problem here.

Runtime Evolutions

We need to make some changes in Nuxeo Runtime :

• for Nuxeo IDE and bundle blacklisting
• for Nuxeo deployment with remote Bundle repository (cf Roadmap slides)

For now, we just need to prepare the Nuxeo Runtime deployment model to be sure that we will be able to plug the necessary items.

• backlisting bundles
• adding bundles from IDE
• hook to download bundles from a remote server before deployment
• check servlet hook (the goal being to remove the deployment-fragment processing)
  • resource loader for WAR
  • contributing Servlets, Filters, Listener via extension points

Big Files and Networking Issues

The issues with Big files in Nuxeo are of 2 types:

• using a Thread and TX on the server side during a long upload / download process
• huge bandwidth usage when several clients download the same asset from a central server

These problems exist in a CAP / DM, but are more visible in DAM when the typical use case is to manipulate big pictures and videos.

Upload / Download Optimization

The typical solution is to use a reverse proxy for buffering upload / download.
This allows to reduce the request time on the Nuxeo side and to free up resources as quickly as possible.

Nginx seems to have standard modules for this, but this is probably available for Apache2 too.

Deported Download

A common use case reported by customers: several users from the same physical site have to access the assets stored on a central server and their network is slow.

The idea would be to download big file from the central server only the first time and use a “local cache” other times.

There seems to be a common pattern for that called X-SendFile :

• central server returns a specific header rather than the actual file
• the reverse proxy intercept this header and use it to do an internal redirect and serve the file from an other location

This pseudo protocol seems to be implemented for Nginx, Lighhtpd and Apache2.

To make it work for Nuxeo, we would need to :
– tweak download servlet to return only the header
– have a decentralized BlobStore with a simple download servlet (no security, download by Digest should be enough)
– have the reverse proxy tag (i.e. add header) the requests to send to Nuxeo server the URL of the local download server

I hope you liked what you just read :) Don’t hesitate to comment here or on our brand new community page on G+.

The post [Nuxeo Tech Report] News from the Developer Front #2 appeared first on Nuxeo Blog.

Hi guys, we have a bug day coming up next Monday, January 7. If you want to contribute, there are many ways to do so!
For instance, if you have found a bug, and did not find a Jira ticket mentioning it, feel free to open one. The bugs related to Nuxeo Platform are to be filed under NXP, and the bugs related to Nuxeo Studio, NXS. All of the bugs we’ll be working on are tagged with ‘bugday‘.

Now, if you’re a developer and you want to help, this is a nice opportunity to give back to the Nuxeo community. You can take tickets tagged with bugday and then submit the fix using GitHub’s pull request. If you’re experiencing difficulties while fixing bugs, you can ping us on our IRC channel (#nuxeo on Freenode).

And, if you are in Paris, you can join us at the office and enjoy a post bug day beer :D

The post Next Nuxeo Bug day scheduled Monday, January 7 appeared first on Nuxeo Blog.

Enabling preview for attachments

Today we have a question about document preview. The default behaviour of the JSF app only gives you the preview of the main file. You don’t have preview for the attachments. But the good news is that technically, it relies on a REST API, so it’s pretty easy to call. Here’s an example from the default preview window:

http://localhost:8080/nuxeo/restAPI/preview/default/2dc340b4-5a14-4fd9-afdb-dc4c1e09830e/default/?blobPostProcessing=true

To understand how this works, let’s take a look at the documentation. Since it’s quite old, it uses the restlet. So in Nuxeo Explorer you have to search for the restlet extension point.

The format of the URL is as follow:

        <urlPattern>/preview/{repo}/{docid}/{fieldPath}/</urlPattern>

So I have to find the right fieldPath to access my attachment. It’s stored in the schema files, as the first element of the files list (usual xpath would be files:files/1/file). Here’s what it looks like:

http://localhost:8080/nuxeo/restAPI/preview/default/2dc340b4-5a14-4fd9-afdb-dc4c1e09830e/files:files-0-file/
Note that the slash in the fieldPath has to be replaced by a dash.

Now you can have a preview of any file. The choice is up to you as to where/how to display it. I like the fancybox. I think it looks great and it’s really easy to set up. Here’s just a quick example:

  <nxu:inputList value="#{currentDocument.files.files}"  id="files_input" model="model">
    <a4j:commandLink ajaxSingle="true" ignoreDupResponses="true" requestDelay="100"
      onclick="javascript:showFancyBox('restAPI/preview/default/#{currentDocument.id}/files:files-#{model.rowIndex}-file/');">
      Preview
    </a4j:commandLink>
  </nxu:inputList>

Also something that you need to know: the HTML preview of any file is computed using converters. So if for some reason you cannot preview a file, you need to make sure there’s an HTML converter for it. We try to have as many as possible by default, but some file types are not covered (usually it’s because they are proprietary, undocumented file formats).

The post [Q&A Friday] Enabling Preview For Document Attachments appeared first on Nuxeo Blog.

It’s Bug Day Pizza time!

As you may have noticed, yesterday was Bug Day for the Nuxeo Community. This means we set aside a full day where we worked together to clean up and fix as many bugs as possible. They are listed in our issue tracker, JIRA. Here’s the list of the 56 bugs we fixed yesterday.

I think my ‘favorite’ bug was the one about concurrency. Benoît discovered it during a Funkload writer bench (because he rocks).

Kudos to our good friend Nelson Silva from Portugal who did several pull requests and fixed bugs with the Nuxeo team. Thanks Nelson!

Another big thanks to Simon Poirier for his contribution to nuxeo-platform-mail.

The post January 2013 Bug Day Results appeared first on Nuxeo Blog.

Nuxeo Platform 5.6.0-HF08 is out and available for Nuxeo Connect clients from the Admin Center or Nuxeo Marketplace. It’s also available in the maintenance branch of the source code tree. You can find the complete release note as usual on Jira.

And now for the main corrections coming with this release:
– Fix relation author when a user creates a relation
– Fix Delete action available for documents from sections
– Fix saving an NXQL query with quotes in the Quick Search gadget
– Fix tasks from previous route instance not to display in the Past tasks view
– Fix lifecycle state when editing an approved document
– Fix DocumentRouteInstancesRoot failure when first document found at the root is not a Domain
– Fix error after a user approves a document which he usually can’t access
– Fix Social workspace login validation URL that redirects to an error message
– Don’t display workflow related documents in search results
– Power users are now able to change the user’s groups from the Edit tab
– Fix NXQL failure on backslash at end of string
– Fix hot reload of content views
– Fix error when computing a virtual user and the group directory is not available
– Document creation form sets focus on title for every browser
– Fix ContentView predicate IN with integer list
– Make ConnectionFactoryImpl init thread-safe
– Fix ContextMenu initialization after Ajax refresh
– Fix workflow cancellation
– Fix incorrect lifecycle state after publishing
– Fix emails encoding to UTF-8 for html mails sent with the SendMail operation

A big thanks to our Support Team for doing this!
* Note that major versions of the Nuxeo Platform are openly available for download. Hot fixes are available for Nuxeo Connect clients as part of their subscription package.

The post [Nuxeo Release] Nuxeo Platform 5.6.0-HF08 is out appeared first on Nuxeo Blog.

The Action registry

The new Category attribute

Don’t forget that if you find bugs in Nuxeo Studio, you can report them on our Jira, in the appropriate project NXS.

I would also like to clarify the naming scheme of Studio version. The first and second number are used for major improvements. The third one is use for maintenance, bug fixing etc..

The post [Nuxeo Release] Nuxeo Studio 2.10 is out! appeared first on Nuxeo Blog.

Welcome to the second part of the file metadata blog post. Last time I explained the issues I wanted to address and started to code a service for that. Today I’ll try to show you when and how to do the actual mapping.

First comes the “When.” We need to update metadata each time the document is about to be modified or created. So I need a Listener with the following configuration:

<component name="org.nuxeo.metadata.listener.contrib.MetadataListener">

  <extension target="org.nuxeo.ecm.core.event.EventServiceComponent"
    point="listener">

    <listener name="metadatalistener" async="false" postCommit="false"
      class="org.nuxeo.metadata.MetadataListener" order="140">
      <event>aboutToCreate</event>
      <event>beforeDocumentModification</event>
    </listener>
  </extension>

</component>

I’ve chosen to listen to aboutToCreate and beforeDocumentModification instead of the common documentModified or documentCreated. This way we don’t have to save document again, which could trigger another documentModified hence a loop.

The listener itself is simple. It just calls the mapping service. This is where all the logic is. This way you can call it from somewhere else like an importer or from an operation.

public class MetadataListener implements EventListener {

    public final BlobsExtractor blobExtractor = new BlobsExtractor();

    public void handleEvent(Event event) throws ClientException {

        if (event.getContext() instanceof DocumentEventContext) {
            DocumentEventContext context = (DocumentEventContext) event.getContext();
            DocumentModel doc = context.getSourceDocument();
            CoreSession coreSession = context.getCoreSession();
            try {
                FileMetadataService fms = Framework.getService(FileMetadataService.class);
                fms.mapMetadata(doc, coreSession);
            } catch (Exception e) {
                throw new RuntimeException(e);
            }
        }

    }
}

Now that we took care of the “When,” we can handle the “How.”

Take a look at the mapMetadata method from the service. Here you want to know if there is a mapper declared for the given document. The first thing to do is get the main blob from the document using the BlobHolder adapter. Then using the document type and the blob mime-type, we compute the ID used to store mapping in the blobHolderMapping map. If there’s a registered mapper, we jump to the mapProperties method. Once the main blob is handled, we go through the different mapping where an xPath was specified. Again if there’s a registered mapper, we jump to mapProperties.

    @Override
    public void mapMetadata(DocumentModel doc, CoreSession coreSession)
            throws ClientException, IOException {
        BlobHolder bh = doc.getAdapter(BlobHolder.class);
        Blob blob = bh.getBlob();
        String blobMimeType = blob.getMimeType();
        String docType = doc.getType();
        String bhId = docType + blobMimeType;
        MetadataMappingDescriptor bhMapper = bhMapping.get(bhId);
        if (bhMapper != null) {
            mapProperties(doc, blob, bhMapper, coreSession);
        }

        Map<String, Map<String, MetadataMappingDescriptor>> nxPathDocMapping = nxPathMapping.get(docType);
        if (nxPathDocMapping != null) {
            for (String nxPath : nxPathDocMapping.keySet()) {
                Blob b = (Blob) doc.getPropertyValue(nxPath);
                if (b != null) {
                    String bMimeType = b.getMimeType();
                    bhMapper = nxPathDocMapping.get(nxPath).get(bMimeType);
                    if (bhMapper != null) {
                        mapProperties(doc, b, bhMapper, coreSession);
                    }
                }
            }
        }
    }

    private void mapProperties(DocumentModel doc, Blob blob,
            MetadataMappingDescriptor metadataMappingDescriptor,
            CoreSession coreSession) throws ClientException, IOException {
        String mapper = metadataMappingDescriptor.getMapperId();
        MetadataMapper mapperInstance = mappers.get(mapper);
        mapperInstance.mapProperties(doc, blob, metadataMappingDescriptor,
                coreSession);
    }

The mapProperties is again very simple. It gets a mapper instance from the mapping descriptor and executes the mapProperties. This is where the mapping happens. It has a really simple contract. It must implement the MetadataMapper interface which has only one method. It makes it easier to implement mapper based on another library than ExifTool or Tika.

These are the two libraries I wanted to try. They are both very powerful for extracting metadata from a file. But I like ExifTool better when it comes to writing metadata to a file. Let’s take a look at the ExifTool mapper. It’s straightforward – there’s no complicated stuff here. It might be interesting if you’ve never used the CommandLineExecutorService.

package org.nuxeo.metadata;

import java.io.File;
import java.io.IOException;
import java.io.Serializable;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.codehaus.jackson.JsonNode;
import org.codehaus.jackson.map.ObjectMapper;
import org.nuxeo.ecm.core.api.Blob;
import org.nuxeo.ecm.core.api.ClientException;
import org.nuxeo.ecm.core.api.CoreSession;
import org.nuxeo.ecm.core.api.DocumentModel;
import org.nuxeo.ecm.core.api.blobholder.BlobHolder;
import org.nuxeo.ecm.core.api.impl.blob.FileBlob;
import org.nuxeo.ecm.core.api.model.Property;
import org.nuxeo.ecm.core.api.model.PropertyException;
import org.nuxeo.ecm.core.storage.sql.coremodel.SQLBlob;
import org.nuxeo.ecm.core.utils.BlobsExtractor;
import org.nuxeo.ecm.platform.commandline.executor.api.CmdParameters;
import org.nuxeo.ecm.platform.commandline.executor.api.CommandAvailability;
import org.nuxeo.ecm.platform.commandline.executor.api.CommandLineExecutorService;
import org.nuxeo.ecm.platform.commandline.executor.api.CommandNotAvailable;
import org.nuxeo.ecm.platform.commandline.executor.api.ExecResult;
import org.nuxeo.runtime.api.Framework;
import org.nuxeo.runtime.services.streaming.FileSource;
import org.nuxeo.runtime.services.streaming.StreamSource;

public class ExifToolMapper implements MetadataMapper {

    public static final Log log = LogFactory.getLog(ExifToolMapper.class);

    public final BlobsExtractor blobExtractor = new BlobsExtractor();

    @Override
    public void mapProperties(DocumentModel doc, Blob blob,
            MetadataMappingDescriptor mappingDescriptor, CoreSession session)
            throws ClientException, IOException {
        String nxPath = mappingDescriptor.getNxpath();
        if (nxPath != null && !"".equals(nxPath)) {
            Property p = doc.getProperty(nxPath);
            if (p != null) {
                if (p.isDirty()) {
                    readProperties(doc, blob, mappingDescriptor, session);
                    // ignore writeChangestoFile
                    return;
                } else {
                    writeChangesToFile(doc, blob, mappingDescriptor, nxPath,
                            session);
                }
            }
        } else {
            // No specified XPath, we assume it's the bh mapper
            // is the property holding the bh dirty?
            try {
                for (Property prop : blobExtractor.getBlobsProperties(doc)) {
                    if (prop.isDirty()) {
                        Blob b = prop.getValue(Blob.class);
                        if (b.equals(blob)) {
                            // we found the property in the BlobHolder
                            readProperties(doc, blob, mappingDescriptor,
                                    session);
                            return;
                        }
                    }
                }
                writeChangesToFile(doc, blob, mappingDescriptor, null, session);
            } catch (Exception e) {
                throw new IOException(
                        "Error while extracting dirty blob properties");
            }
        }

    }

    private void writeChangesToFile(DocumentModel doc, Blob blob,
            MetadataMappingDescriptor mappingDescriptor, String nxPath,
            CoreSession session) throws ClientException, IOException {
        StringBuilder sb = new StringBuilder();
        for (PropertyItemDescriptor property : mappingDescriptor.getProperties()) {
            Property p = doc.getProperty(property.getXpath());
            if (p.isDirty() && property.getPolicy().equals("sync")) {
                // write this property to the file
                sb.append('-');
                sb.append(property.getMetadataName());
                sb.append('=');
                sb.append("\\\\'");
                sb.append(p.getValue());
                sb.append("\\\\'");
                sb.append(' ');
            }
        }
        String tagList = sb.toString();

        CommandLineExecutorService cles;
        try {
            cles = Framework.getService(CommandLineExecutorService.class);
        } catch (Exception e) {
            throw new RuntimeException(
                    "ComandLineExecutorService was not found.", e);
        }

        CommandAvailability ca = cles.getCommandAvailability("exiftool-write");
        if (!ca.isAvailable()) {
            log.warn("Attempted to use exiftool but did not find it. ");
            return;
        }
        File file = makeFile(blob);
        CmdParameters params = new CmdParameters();
        params.addNamedParameter("tagList", tagList);
        params.addNamedParameter("inFilePath", file);
        try {
            cles.execCommand("exiftool-write", params);
        } catch (CommandNotAvailable e) {
            throw new RuntimeException("Command exiftool is not available.", e);
        }
        Blob fileBlob = new FileBlob(file);
        if (nxPath == null) {
            BlobHolder bh = doc.getAdapter(BlobHolder.class);
            bh.setBlob(fileBlob);
        } else {
            doc.setPropertyValue(nxPath, (Serializable) fileBlob);
        }
    }

    private void readProperties(DocumentModel doc, Blob blob,
            MetadataMappingDescriptor mappingDescriptor, CoreSession session)
            throws IOException, PropertyException, ClientException {
        CommandLineExecutorService cles;
        try {
            cles = Framework.getService(CommandLineExecutorService.class);
        } catch (Exception e) {
            throw new RuntimeException(
                    "ComandLineExecutorService was not found.", e);
        }

        CommandAvailability ca = cles.getCommandAvailability("exiftool-read");
        if (!ca.isAvailable()) {
            log.warn("Attempted to use exiftool but did not find it. ");
            return;
        }
        CmdParameters params = new CmdParameters();
        File file = makeFile(blob);
        params.addNamedParameter("inFilePath", file);
        try {
            ExecResult er = cles.execCommand("exiftool-read", params);
            if (!er.isSuccessful()) {
                log.error("There was an error executing the following command: "
                        + er.getCommandLine());
                return;
            }
            // check facets availability
            String[] requiredFacets = mappingDescriptor.getRequiredFacets();
            for (String facet : requiredFacets) {
                if (!doc.hasFacet(facet)) {
                    doc.addFacet(facet);
                }
            }

            StringBuilder sb = new StringBuilder();
            for (String line : er.getOutput()) {
                sb.append(line);
            }
            String jsonOutput = sb.toString();
            ObjectMapper jacksonMapper = new ObjectMapper();
            JsonNode jsArray = jacksonMapper.readTree(jsonOutput);
            JsonNode jsonObject = jsArray.get(0);
            for (PropertyItemDescriptor property : mappingDescriptor.getProperties()) {
                JsonNode node = jsonObject.get(property.getMetadataName());
                if (node == null) {
                    // try fallback
                    for (String fallbackMetadata: property.fallbackMetadata) {
                        node = jsonObject.get(fallbackMetadata);
                        if (node != null) {
                            break;
                        }
                    }
                }
                if (node != null) {
                    String metadata = node.getValueAsText();
                    if (metadata != null && !"".equals(metadata)) {
                        Property p = doc.getProperty(property.getXpath());
                        p.setValue(metadata);
                    }
                }
            }

        } catch (CommandNotAvailable e) {
            throw new RuntimeException("Command exiftool is not available.", e);
        }
    }

    protected File makeFile(Blob blob) throws IOException {
        File sourceFile = getFileFromBlob(blob);
        if (sourceFile == null) {
            String filename = blob.getFilename();
            sourceFile = File.createTempFile(filename, ".tmp");
            blob.transferTo(sourceFile);
            Framework.trackFile(sourceFile, this);
        }
        return sourceFile;
    }

    protected File getFileFromBlob(Blob blob) {
        if (blob instanceof FileBlob) {
            return ((FileBlob) blob).getFile();
        } else if (blob instanceof SQLBlob) {
            StreamSource source = ((SQLBlob) blob).getBinary().getStreamSource();
            return ((FileSource) source).getFile();
        }
        return null;
    }
}

Let us know what you think of this. We don’t have many mappings declared right now, so any help is welcome. There might be some refactoring going on before the 5.7 release. I’ll keep you posted. See you Friday or on our G+ community page if you want to talk Nuxeo :-)

The post [Monday Dev Heaven] Extract Metadata from Content File Attachments #2 appeared first on Nuxeo Blog.

Notification of impending expiration of a document

Here’s a very practical question from blaszta:

How can I set an email notification about the impending expiration of a document (let’s say a 3 year contract leased document), 1 month before it expires?

Let’s break this in two parts. First, get the contract that expires in 1 month. Then send the email.

Query the contracts

If you’re looking for specific document like a contract, there’s a good chance you have an easy way to identify one using its type or some metadata. Let’s say all contract documents are of type Contract. If I want all of them I can do the following query:

Select * From Contract

This gets me all the contract documents, even the deleted ones or different versions of the same document. To avoid this we can add the following parameters:

Select * From Contract WHERE ecm:isCheckedInVersion = 0 AND
 ecm:currentLifeCycleState != 'deleted'

Now you want to add something that gets you only Contracts that expire in 30 days, no more, no less. I’ll let you compute the date yourself :) – you get the idea.

Select * From Contract WHERE ecm:isCheckedInVersion = 0 AND
 ecm:currentLifeCycleState != 'deleted' AND dc:expired = DATE '2007-02-15'

If you want more information about queries in Nuxeo, take a look at the NXQL documentation.

Now we know we can get those documents, we can move on to the second part.

Send the email

We need to send the notifications only once a day. For that we can use an event listener coupled with the scheduler service. We’ll send the event notifContract one time per day every day. Then we can add a listener to the notifContract event that does the query and sends the notifications.

Here’s a contribution to send a notifContract event every day at 1am.

<?xml version="1.0"?>
<component name="org.nuxeo.sample.scheduler.notifContract">
  <extension
    target="org.nuxeo.ecm.core.scheduler.SchedulerService"
    point="schedule">
    <schedule id="notifContractScehdulerId">
      <username>Administrator</username>
      <eventId>notifContract</eventId>
      <eventCategory>default</eventCategory>
      <!-- every day at 1 am -->
      <cronExpression>0 0 1 * * ?</cronExpression>
    </schedule>
  </extension>
</component>

And here’s the associated listener:

<?xml version="1.0"?>
<component name="org.nuxeo.sample.listener.contrib.notifContract">
  <extension target="org.nuxeo.ecm.core.event.EventServiceComponent"
    point="listener">
    <listener name="notifContractListener" async="true" postCommit="false"
      class="org.nuxeo.sample.NotifContractListener" order="140">
      <event>notifContract</event>
    </listener>
  </extension>
</component>

Now we have to write the NotifContractListener class so that it queries contracts and sends an email for each contract. There are multiple ways to send the email. You could handle everything yourself, which means gathering users that need notifications for each document, then rendering and sending an email to everyone of them. Or you could use the notification service provided by Nuxeo, which is much simpler and what we’re going to do.

We’re going to send a new event for each document from the query, with the document attached. Then we’ll make this event available as a usual notification, like the ones you find on the Alert tab.

First comes the listener implementation, where we execute the query and send an event for each document returned by the query.

package org.nuxeo.sample;

import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Date;

import org.nuxeo.ecm.core.api.ClientException;
import org.nuxeo.ecm.core.api.CoreSession;
import org.nuxeo.ecm.core.api.DocumentModel;
import org.nuxeo.ecm.core.api.DocumentModelList;
import org.nuxeo.ecm.core.event.Event;
import org.nuxeo.ecm.core.event.EventListener;
import org.nuxeo.ecm.core.event.EventService;
import org.nuxeo.ecm.core.event.impl.DocumentEventContext;
import org.nuxeo.runtime.api.Framework;

/**
 * @author ldoguin
 */
public class ContractNotifListener implements EventListener {

    private static final String QUERY_CONTRACTS = "Select * From Contract WHERE ecm:isCheckedInVersion = 0 AND ecm:currentLifeCycleState != 'deleted' AND dc:expired = DATE '%s'";

    public void handleEvent(Event event) throws ClientException {
        CoreSession coreSession = event.getContext().getCoreSession();
        Calendar expirationDate  = Calendar.getInstance();
        expirationDate.add(Calendar.DATE, 30);
        Date now = expirationDate.getTime();
        String date = new SimpleDateFormat("yyyy-MM-dd").format(now);
        String query = String.format(QUERY_CONTRACTS, date);
        DocumentModelList contracts = coreSession.query(query);

        EventService eventService;
        try {
            eventService = Framework.getService(EventService.class);
            for (DocumentModel contract : contracts) {
                DocumentEventContext ctx = new DocumentEventContext(
                        coreSession, coreSession.getPrincipal(), contract);
                Event contractExpiredEvent = ctx.newEvent("contractExpired");
                eventService.fireEvent(event);
            }
        } catch (Exception e) {
            throw new RuntimeException("could not get the EventService", e);
        }
    }

}

Now that an event is sent when a contract expires in 30 days, we need to warn the user. It’s a simple contribution to the notification extension point.

<?xml version="1.0"?>
<component
  name="org.nuxeo.sample.contract.notifcontrib">

  <extension
    target="org.nuxeo.ecm.platform.ec.notification.service.NotificationService"
    point="notifications">
    <notification name="Contract Expired" channel="email" enabled="true" availableIn="Workspace"
      autoSubscribed="false" template="contractExpired" subject="Contract expired" label="label.nuxeo.notifications.contractExpired">
      <event name="contractExpired"/>
    </notification>
  </extension>

  <extension
    target="org.nuxeo.ecm.platform.ec.notification.service.NotificationService"
    point="templates">
    <template name="contractExpired" src="templates/contractExpired.ftl" />
  </extension>

</component>

And that’s it, you’re now set up to send an email alert 30 days before a document expires.

The post [Q&A Friday] Document Expiration Notification appeared first on Nuxeo Blog.

You are reading my first Nuxeo blog.

Writing blogs is a big responsibility. Because, when you think about it, well, it is read. And this is serious stuff you know, when people read your writing. This is why when you write a public blog, you should avoid writing silly things. At least, if you write silly things, it must be done without realizing it. And without reviewers, if any, to realize it.

Oh, by the way, I forgot to introduce myself, which is not very polite and would have displeased my mother (please don’t let her know). To summarize, my name is Thibaud Arguillere (Thibaud is the first name), and I joined Nuxeo last December. I am working with the US team and am based in New York. “Presales Engineer & Technology Evangelist” is my title. It looks serious and makes me feel important. For more details about my professional life, you can check my LinkedIn account.

To build my first Nuxeo demos, I’m using Nuxeo Studio. I’ll share my experience throughout this series of articles modestly entitled “Nuxeo Studio Rocks”.

Let’s go!

You already know this: Nuxeo is the best platform available to build content-centric business applications(1). Even more, it is a truly open source platform (note to self: good topic for a blog, “What Does Truly Mean in “Truly Open Source Platform”?”)

One can easily download the platform, install it in 2 minutes and then use it with its main modules (the most commonly used module as of today is DM, Document Management).

Installing Nuxeo with the Wizard

However, most of the time you need to extend the platform, to configure your applications and, most importantly, to add logic. Business logic. Most of the parameters that Nuxeo loads when your application starts, or loads only as needed at runtime (for example, generating a form for the browser), are XML-based. There is a large number of tags that you can use, covering a wide area, from UI customization (login dialog, buttons, …) to complex workflow definition, to defining forms for creation / visualization / modification / queries, and more.

Everybody just loves XML(3). It’s standard, it’s easy to read(4), every application understands it, and you can find more XML parsers than developers in the whole universe, whatever the language. Right?

But the platform is so rich that it can make this kind of customization a complex process, even if all is well documented. As for me, I much prefer to drag and drop workflow nodes in Studio rather than write XML to the terminal with vi. And when it is time to define an XHTML form, hey, let’s face it: dropping an attribute, binding it to a widget, letting Studio do the job of translating the form to XHTML is definitely easier. And more fun.

Essentially, Studio is an XML Editor. A GSXE more precisely: Graphical Specialized XML Editor(5). You create your application, Studio then generates the appropriate .jar files that your server (development, test, production) can download. This .jar contains the tags (XML, XHTML) that define your application.

In this screenshot, we defined an Automation Chain in the Studio.

Once the server gets the project, Studio generates all the configuration elements and files for me, and inside the OSGI-INF/extensions.xml file, I can find my chain, line #2,055:

Taking these screenshots made me confirm “oh yes! I sure prefer to drag-drop rather than writing thousands of lines of XML”.

Now that you know that Studio is an XML tool, we’ll be using it in the next article, “Nuxeo Studio Rocks: Part #1”.

Have a nice day, morning, evening or night, depending on your location.

(1) It’s not me saying this, it’s our customers. (2)

(2) Actually, well, I say it too

(3) Don’t you?

(4) Good glasses, zoom, syntax coloring recommended

(5) Just invented it. We should organize a contest about how you pronounce GSXE.

The post Nuxeo Studio Rocks – Part 0 appeared first on Nuxeo Blog.

Webinar Slide Example

Today we’re going to play with the templating module. It lets you render a document using a template. You can easily install it through our Marketplace. It comes with a bunch of examples to give you an idea of what’s doable. You will also find information in our user guide.

A live example is the last Webinar slide deck. It was written in a Markdown Note document and rendered using a DeckJS based template.

If you want to try it now it’s available in the nuxeo-template-rendering-deckjs bundle. Just copy the jar in the plugins folder and the template and its example should be automatically loaded. Be aware that the pdf conversion will only work if you have PhantomJS (version >= 1.8) installed on your server.

The template document is a regular WebTemplate. You need to import JavaScripts, StyleSheets, images, etc. as document attachments. Then where you usually have a path like:

<link rel="stylesheet" href="./core/deck.core.css">

You need to replace it by:

  <link rel="stylesheet" href="${jaxrs.getResourceUrl("deck.core.css")}">

This is how you load document attachments during rendering. We’ve also added a custom JavaScript file to do preprocessing of the Markdown content. It creates the different sections and transforms divs like:

<div class="picture" title="app-lifecycle.png">Application LifeCycle schema
</div>

Into this:

<div class="split-box split-37">
  <img src="/nuxeo/site/templates/doc/e9b1cbeb-d0dc-450b-816f-de931f557ce9/resource/NuxeoWorld2K12HtmlSlides/long-road.jpg">
</div>

Here’s our JavaScript preprocessor:

function buildSections(baseUrl) {
var titles = $("h1[id!='slideDeckTitle']");

for (var i = titles.length-1; i >= 0; i--) {
 var t = $(titles[i]).html();
 var tid = t.replace(new RegExp(" "), "-").toLowerCase();
 var slide = $("<section></section>");
 slide.attr("id", tid);
 slide.addClass("slide");
 slide.append("<h3>" + t + "</h3>");
 var slideContent =$("<div></div>");
 var subsections = $(titles[i]).nextUntil("h1");
 for ( var j = 0; j < subsections.length; j++) {
    if (subsections[j].id=='thank-you') {
       break;
    }
    slideContent.append(subsections[j]);
 }
 slide.append(slideContent);
 processSlideLayout(slideContent, slide, baseUrl);

 $("#nuxeo-slides").after(slide);
}
$("h1[id!='slideDeckTitle']").remove();
}

function processSlideLayout(slideContent, parent, baseUrl) {

  var pictures = slideContent.find("div.pictureInline");
  if (pictures.length>=0) {
    for ( var i = 0; i < pictures.length; i++) {
       var img = $("<img/>");
       img.attr("src",baseUrl + $(pictures[i]).attr("title"));
       $(pictures[i]).append(img);
    }
  }

  pictures = slideContent.find("div.picture");
  var pSize = 37;
  if (pictures.length==0) {
    pictures = slideContent.find("div.pictureBig");
    if (pictures.length==0) {
      pictures = slideContent.find("div.pictureLarge");
      if (pictures.length==0) {
	return;
      } else {
        pSize=75;
      }
    } else {
      pSize=50;
    }
  }

  slideContent.addClass("split-box").addClass("split-63");
  var picHolder =$('<div class="split-box"></div>');
  if (pSize==50) {
    slideContent.addClass("split-50");
    picHolder.addClass("split-50");
  }
  else if (pSize==75) {
    slideContent.addClass("split-25");
    picHolder.addClass("split-75");
  }
  else {
    slideContent.addClass("split-63");
    picHolder.addClass("split-37");
  }

  for ( var i = 0; i < pictures.length; i++) {
     var img = $("<img/>");
     img.attr("src",baseUrl + $(pictures[i]).attr("title"));
     picHolder.append(img);
     $(pictures[i]).remove();
  }
  parent.append(picHolder);
}

function preProcessHtml(url) {
 buildSections(url);
}

As you can see this script makes it mandatory to have a h1 tag or an element with id ‘slideDeckTitle’. This is where the script will start its processing. It stops the processing once it finds a section with the ‘thank-you’ id.

Now that the template is setup, you can create a Note in Markdown, associate it to your template and see the results using the webview rendition or the render button from the template tab.

A small word about rendition vs. template rendering. This is not the same thing, even if the name is close. A rendition is :

• a name
• a label
• a class implementing the RenditionProvider interface

So you can have rendition based on a simple converter, on an operation or even a template rendering. More details about rendition are available on the rendition module ReadMe.

Anyway here's a sample for our DeckJS template:

# 2013 Roadmap: Strategy

#### Extend the platform approach

 - Continue to improve the infrastructure
 - Manage the complete application life-cycle

#### Prepare Nuxeo Platform 6.0

- Prepare an infrastructure update

<div class="picture" title="app-lifecycle.png">Application LifeCycle schema
</div>

But what you usually want to do when creating slides is to get a PDF version of it. So we need to convert that HTML file to a PDF. To do that you can select the output format available in the template configuration. Problem is the default PDF converter won't work on out DeckJS file. It won't use the JavaScript during the conversion. So you will end up with a weird one page PDF.

This means we have to use a custom output format. There's an extension point for this. You can specify a mime type and an operation chain id. If there is no operation chain specified, a simple convert operation is called using the mime type as parameter. If there is a chain, we simply call that operation chain.

  <extension target="org.nuxeo.template.service.TemplateProcessorComponent" point="outputFormat">
    <outputFormat id="pdf" label="PDF" mimetype="application/pdf"/>
  </extension>

So we can add a custom operation chain that calls a PhantomJS based converter. PhantomJS is a headless WebKit with JavaScript API. We made a little script that removes some CSS class that we don't need in our PDF, then take a PNG screencap of each slide and put them together as a single PDF file.

  <extension target="org.nuxeo.template.service.TemplateProcessorComponent"
    point="outputFormat">
    <outputFormat id="deckJsToPDF" label="PDF (from DeckJS)"
      chainId="deckJs2PDF" mimetype="application/pdf" />
  </extension>

The chain simply call the Blob.DeckJSToPDF operation. It's not a simple conversion, there is a little trick. When you get the rendered HTML file, all the resources are relative to the server, like this:

  <link rel="stylesheet" href="/nuxeo/site/templates/doc/9b58adaf-68bc-4cda-ae6b-2503c05b610b/resource/NuxeoWorld2K12HtmlSlides/deck.core.css">

So PhantomJS has to go through usual Nuxeo authentication. To avoid that, we can start by removing the parent path, leaving only the filename. This way PhantomJS will be looking for the resources as local files instead of trying to fetch them on the server. Once this is done, we need to make those resources available locally. It means we need to write all doc attachments from the template document and the rendered document to the filesystem. Here's the operation that does this work:

/*
 * (C) Copyright 2006-2012 Nuxeo SA (http://nuxeo.com/) and contributors.
 *
 * All rights reserved. This program and the accompanying materials
 * are made available under the terms of the GNU Lesser General Public License
 * (LGPL) version 2.1 which accompanies this distribution, and is available at
 * http://www.gnu.org/licenses/lgpl.html
 *
 * This library is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
 * Lesser General Public License for more details.
 *
 * Contributors:
 *     ldoguin
 */
package org.nuxeo.template;

import java.io.File;
import java.io.FileWriter;
import java.io.IOException;

import org.apache.commons.io.FileUtils;
import org.apache.commons.io.IOUtils;
import org.nuxeo.ecm.automation.OperationContext;
import org.nuxeo.ecm.automation.core.Constants;
import org.nuxeo.ecm.automation.core.annotations.Context;
import org.nuxeo.ecm.automation.core.annotations.Operation;
import org.nuxeo.ecm.automation.core.annotations.OperationMethod;
import org.nuxeo.ecm.core.api.Blob;
import org.nuxeo.ecm.core.api.DocumentModel;
import org.nuxeo.ecm.core.api.blobholder.BlobHolder;
import org.nuxeo.ecm.core.api.impl.blob.FileBlob;
import org.nuxeo.ecm.core.convert.api.ConversionService;
import org.nuxeo.ecm.core.convert.cache.SimpleCachableBlobHolder;
import org.nuxeo.template.jaxrs.context.JAXRSExtensions;

@Operation(id = DeckJSPDFOperation.ID, category = Constants.CAT_CONVERSION, label = "Convert a deckJS slide to a pdf", description = "Convert a deckJS slide to a pdf.")
public class DeckJSPDFOperation {

    public static final String ID = "Blob.DeckJSToPDF";

    @Context
    OperationContext ctx;

    @Context
    ConversionService conversionService;

    @OperationMethod
    public Blob run(Blob blob) throws Exception {
        DocumentModel templateSourceDocument = (DocumentModel) ctx.get("templateSourceDocument");
        DocumentModel templateBasedDocument = (DocumentModel) ctx.get("templateBasedDocument");
        String templateName = (String) ctx.get("templateName");

        String workingDirPath = System.getProperty("java.io.tmpdir")
                + "/nuxeo-deckJS-cache/" + templateBasedDocument.getId();
        File workingDir = new File(workingDirPath);
        if (!workingDir.exists()) {
            workingDir.mkdirs();
        }
        JAXRSExtensions jaxRsExtensions = new JAXRSExtensions(
                templateBasedDocument, null, templateName);
        BlobHolder sourceBh = templateSourceDocument.getAdapter(BlobHolder.class);
        for (Blob b : sourceBh.getBlobs()) {
            writeToTempDirectory(workingDir, b);
        }
        BlobHolder templatebasedBh = templateBasedDocument.getAdapter(BlobHolder.class);
        for (Blob b : templatebasedBh.getBlobs()) {
            writeToTempDirectory(workingDir, b);
        }

        String content = blob.getString();
        String resourcePath = jaxRsExtensions.getResourceUrl("");
        content = content.replaceAll(resourcePath, "./");
        File index = new File(workingDir, blob.getFilename());
        FileWriter fw = new FileWriter(index);
        IOUtils.write(content, fw);
        fw.flush();
        fw.close();

        FileBlob indexBlob = new FileBlob(index);
        indexBlob.setFilename(blob.getFilename());
        BlobHolder bh = conversionService.convert("deckJSToPDF",
                new SimpleCachableBlobHolder(indexBlob), null);
        FileUtils.deleteDirectory(workingDir);
        return bh.getBlob();
    }

    private void writeToTempDirectory(File workingDir, Blob b) throws IOException {
        File f = new File(workingDir, b.getFilename());
        File parentFile = f.getParentFile();
        parentFile.mkdirs();
        b.transferTo(f);
    }
}

Then, once the resources are written to the filesystem, we can call the converter based on PhantomJS. The code is available on GitHub if you want to check it out.

The post [Monday Dev Heaven] Creating DeckJS Slides in Markdown appeared first on Nuxeo Blog.

How to add extra files to a document using Content Automation

Today we have someone asking how to attach extra files to a document using Content Automation. So I’m going to do this using nuxeo-automation-client. This is a jar that you can add as a dependency to your Java application and that gives you a nice API to make content automation call to a Nuxeo server.

I’ve written a small example that opens a Content Automation HTTP session to a local Nuxeo server as Administrator and execute an operation that adds a file to an existing document. Those operations are wrapped in the DocumentService. This class hides part of the complexity to make an operation request. You can take a look at our documentation for the complete details. Now here’s the code sample, most of it is explained in the comments:

public class SampleAttachFiles {

    public static void main(String[] args) throws Exception {
        try {
            // Create an instance of the client using the address of your
            // server.
            HttpAutomationClient client = new HttpAutomationClient(
                    "http://localhost:8080/nuxeo/site/automation");
            // Open a session as Administrator with password Administrator
            Session session = client.getSession("Administrator",
                    "Administrator");
            // let's assume you now the id of the document
            String docId = "0e8d31c3-eef1-43fe-a575-7d63705ecbbf";
            // and the path to the file to attach
            File file = new File("/path/to/your/file");
            // The DocumentService will give you some nice shortcut to the most
            // common operations
            DocumentService rs = session.getAdapter(DocumentService.class);
            // Create a document Ref object with the id of the document
            DocRef docRef = new DocRef(docId);
            // Create a blob object from the file to upload
            Blob blob = new FileBlob(file);
            // Use DocumentService to attach the blob. Giving files:files as
            // argument will add the blob to the existing attachment
            rs.setBlob(docRef, blob, "files:files");
            // This will replace the content of the firt attachement
            rs.setBlob(docRef, blob, "files:files/0/file");
            // If you do that, do not forget to change the filename too
            rs.setProperty(docRef, "files:files/0/filename", "NewName");
            client.shutdown();
        } catch (RemoteException e) {
            e.printStackTrace();
            System.out.println(e.getRemoteStackTrace());
        }
    }
}

The post [Q&A Friday] How to add extra files to a document using Content Automation appeared first on Nuxeo Blog.

Today I’m particularly happy to announce that an open source add-on from our partner Astone Solutions is available on the Nuxeo Marketplace. It’s an Outlook plugin that lets you save email from your inbox to any folder in Nuxeo. The source code is on GitHub.

Outlook Add-on for Nuxeo

It comes in two parts: server and client. The server part can be installed as usual through the Marketplace. The client comes as an executable with an MSI file. Simply launch the executable to install, but make sure it’s in the same folder as the MSI.

Configuration is a simple panel where you give the URL of your Nuxeo server, the login and the password of the user. And that’s all you have to do. Now if you right click on any email, you’ll see a “Send to Nuxeo” menu item. Click on it and you’ll see your server document tree. Select a folder and hit save. Your email is now inside Nuxeo , with all its attachments. Neat huh?

It works with Outlook 2007 and 2010. Here is a short example on Outlook 2007:

As you can see in the video, it’s dead easy to use. So once again, a big thanks to Astone Solutions for their contribution.

The post Save Outlook Email in Nuxeo appeared first on Nuxeo Blog.

Hello everyone! Here is the first Tech Report of the year – as usual taken straight from the meetings of the Nuxeo Dev Team. You’ll see there are many many different things going on right now.

2013 Roadmap

The global roadmap is being slightly updated. You can get the details by watching a replay of our last Roadmap Webinar.

DAM 2

This is the big part of this report. Work on DAM 2 has started and there is a lot to talk about :)

Web application

Progress report

Thomas started rebuilding the DAM web application from the ground last week. The main goals are to align on Layouts and ContentViews so that the DAM view is configurable via Studio.10

The good news is that after 6 days of work, we already have a working DAM view:

• using Layout for page layout
• using Actions, Widgets, and Grid Layout for content
• with support for Ajax
• with support for bookmarkable URLs

THE DAM2 project only contains 2 small beans and few templates. This means we are mostly relying on the Nuxeo CAP infrastructure.

Ajax navigation

DAM2 will be used as a sandbox to improve Ajax integration inside Nuxeo.

This covers several aspects:

REST URL update
This is done with a generic fragment calling HTML5 history.push() in an Ajax rendered panel. This is something we will push into Nuxeo CAP soon.

ContentView Pagination

This requires adding contentViewName, pageIndex and pageSize inside the URL pattern and inside the associated codec. This is being done inside DAM for now, but will likely be integrated inside Nuxeo CAP default codecs.

Switching tabs
This was already prototyped in DM. Now that the infrastructure is ready (via better management of JSF forms), we should be able to integrate this in DAM and then in CAP/DM.

UX

We may want DAM2 to be usable via the keyboard.

The goal is to allow browsing assets in a “natural way”:

• using arrows to navigate
• using PageUp/PageDown for pagination
• using Enter to select an asset

Problems / Issues

Facets and merge

DAM2 displays all documents having the Asset facet. This means the DAM plugin should add the Asset facet to all Picture/Video/Audio documents that already exist. This was a problem because Doc Types didn’t support merge for Facets. This is now fixed.

Faceted search dependency

DAM2 will have a dependency on FacetedSearch, so the DAM package will need to embed faceted search in the Marketplace Package if we want to be able to install DAM on top of CAP (i.e. without DM).

This may require some changes in Faceted Search to be sure it won’t give a funny UI or broken contrib on CAP (i.e. don’t embeded FacetedSearch-DM part ).

Codec

Some work was done on the DAM codec (based on docPath) to be able to manage “null document” case. We need to add this fallback directly in the DocumentPathCodec.

HiddenInFacetedSearch

Faceted Search contributes the HiddenInFacetedSearch facet for the saved search documents.
This facet is used to filter the saved search from the result listing.

We should remove this facet and simply add the HiddenInNavigation facet. As for edit / manage saved search, we should propose a dedicated screen with a dedicated contentView that will simply bypass the HiddenInNavigation filtering.

ContentView, PageProvide and currentDocument

DAM contains a Master/Detail view type:

• ContentView display the master / listing part
• the currentDocument is displayed on the right panel

This means that when doing prev/next page on the ContentView we must update the currentDocument to one being part of the current page.

We will use an Event driven system based on Seam events:
contentViewChanged => update the currentDocument to be in the currentPage

The PageProvider itself cannot do it since it does not have access to Seam JSF.

The current idea is:

• add a contributable listener in AbstractPageProvider
• make ContentView add a Seam-aware listener class (that will fire a Seam event)
• make DAM add a Seam listener that will update the currentDocument according to the currentPage

Drive

Automation changes

Object2JSON

For Drive requirements, Olivier had to add support for native Java Object 2 JSON marshaling.

Complex types

At the same time, Olivier started looking at the complex property limitations in Automation API. The create/update Automation API is really not easy to use to manage complex properties.

Historically this was simply not possible, later a hack was added for the Android Client, but the full fix was never done.

A complete solution includes:

• managing JSON encoded complex properties inside the PropertyMap object (already partially done)
• send the PropertyMap object as a real JSON object when possible (i.e. remove the limitation for \n!)
• update the Java Automation Client to manage complex properties and dirty states

Automation protocol version and non-regression

The changes introduced in Automation Marshaling may at some point impact compatibility.

This is something we ideally don’t want.
If we call Automation V2 the new Automation that manages Objects and Complex properties, Automation Server V2 should be able to handle calls from Automation V1 and V2 clients.

This could be a good point to have a REST (Funkload) test using Automation API that is recorded against 5.6 and check that it still runs against a 5.7 server:

• depending on the result, we will know if we must really manage Automation Version in the protocol via a Client/Server negotiation
• this will be useful for non-regression.

Repository Work

Quota Module

Performance impact

Managing Quotas implies additional checks and additional Write operations in the repository.

In addition, Quotas are mainly useful when there is a lot of data and a lot of users.

These 2 points raise a warning for performance, so Ben did some tests :)

Funload Benchmarking

A Funkload test for document creation was run to compare with and without Quota add-ons.

The results show that adding Quota management:

• reduces scalability by 20%
• increases response time by 25%

Most of the work is done inside Listeners:

• quotaStatsListener is slow 83% of time of all sync listeners
• quotaProcessor 15% of all async listeners

DB processing

There is a high number of calls of quotaStatsListener on document creation:

• for the created document:
  • DOCUMENT_CREATED
  • BEFORE_DOC_UPDATE
  • DOCUMENT_UPDATED
• for each parent:
  • BEFORE_DOC_UPDATE
  • DOCUMENT_UPDATE

3 + depth *2 calls = 11 calls for a creation on depth 4.
The work is done only in BEFORE_DOC_UPDATE and DOCUMENT_CREATED (nothing on DOCUMENT_UPDATE).
The number of SQL UPDATEs to set the size and count is correct (no duplicate commands).

VCS optimization

UUIDs (NXP-4803 uuid for document id)

The branch NXP-4803-db-uuid implements changes for PostgreSQL.

The data migration from varchar to uuid works in 4 steps:

1. sql dump
2. create a new db
3. import a uuid schema
4. import the sql dump

Migration speed on a good PGSQL server is about 2000 docs / s.

A bench is in progress using the CI.
Initial tests were done on octopussy with 600k docs:

• uuid index size is 45% smaller
• index is certainly less bloated after a mass import.

Once the CI job update is done, we should have a bench with big volumes and a diff between the master and the uuid branch.

SQL Server 2012

NXP-9660 support of mssql 2012:

The default collation is case sensitive and there are system tables that cannot be dropped. Changes have been made on the master. Unit tests and funkload tests are ok.
=> we can consider that MSSQL 2K12 is now supported (YAY!)

NXP-10640 Avoid issues on concurrent write:

Switching the transaction level from “SNAPSHOT” to “READ COMMITTED” removes the “Snapshot isolation transaction aborted”.
But reintroduces a deadlock on ACL optimization update. It is not easy to reproduce but happens with importer addon or on a long bench.

One solution is to synchronize the updateReadACL call in SessionImpl.doFlush. This works so far.

Note that the synchronized code prevents having multiple update read ACLs at the same
time (at least for each Nuxeo node), which should be fine for all databases.

Ben will test it with the ondemand-bench job to see if there is performance regression.

The way we handle the updateReadACL for now is just a first step:

• phase 1: add the synchronize java block
• phase 2: run processing in async
• phase 3: provide a cluster-wide lock / sync system.

Clustered index

It looks like MSSQL Server needs to have a clustered index, and by default uses the PK.
=> we should add auto-incremed int columns and mark it as clustered
==> this should not impact any java code and may improve the MSSQL performance!

JDBC and cast

The JDNC driver transfers all string parameters as UTF-8. As a result, there may be cast on the database side when looking at ASCII columns like UUIDS and this makes SQL Server skip the indexes and scan the table.

Async update

Some operations in VCS can start long running transactions. This is typically the case when a change inside the repository triggers a recomputation of ReadACL or Ancestors tables. The worst case is to move a folder on top of a big hierarchy:
– trigger rebuilds ACLs
– trigger rebuilds ancestors

This long running TX leads to 2 problems:

• slow UI and possible TX timeout
• concurrency issue, because on some databases, the tables end up being locked by the update process.

The ideal solution would be:

• optimize ancestors update (ex: for rename)
• run update in async + mono-thread.

Infrastructure

Deployment fragment

The goal is to align a Servlet 3 spec to manage modular web deployment.

DataSource internalization

We recently discovered that the DataSources managed directly by Tomcat (i.e. all except VCS) are not correctly enlisted in transactions. Stephane started to manage DataSources via an extension point system.

Advantages:

• we can correctly enlist in the Tx
• this make the configuration more flexible
• the configuration no longer depends on the application server

Drawbacks:

• Default Tomcat monitoring won’t see our DataSources
• we don’t leverage the application server infrastructure.

We’ll see how we manage this with respect to JBoss, but we’ll try to keep the option of using application server level DataSources, but in the default Tomcat distribution we’ll use Nuxeo “internal DataSources”.

Doing this forces the service that needs to initialize persistence to wait for the DataSource services to be initialized. This forced Stephane to make changes in Nuxeo Runtime to better manage the ApplicationStarted event. This results in better management of the Component LifeCycle:

• we now manage a “started” state (like in the OSGI model :) )
• bundle notification will be usable during reload.

This will be committed in a branch and we’ll wait for QA to validate this.

Tomcat 7

Tomcat 7 support is available in a pending branch. We must merge this branch so that 5.7 will be aligned on Tomcat 7.

Critical section

As you may already have experienced, managing concurrency between several threads trying to create overlapping subtrees in the repository is not easy (NXP-10707).

We already had the issue on several projects where import jobs are using the personal workspace. At some point, 2 import jobs will create the same UserWorkspace (because of MVCC and isolation) but one will fail.

Avoiding this required having a critical section pattern that:

• works cluster-wide (JVM-level locks don’t fix the issue)
• manages Transaction visibility constraints.

Stephane started the work in order to provide some code samples for support.

This code is in the NXP-10707-critical-section branch.

Add-ons

Metadata

The goal is to manage metadata extract / writeback for some file types (pictures, videos). For now, inside Nuxeo we have basic support:
– based on ImageMagic and FFMpeg
– extract only
– maps file metadata to a fixed Nuxeo schema (IPTC, EXIF)
In the last weeks:
– Some work was done for the blog
– define service using an external tool
– manage extract AND writeback
– manage configurable mapping (no need for a fixed schema)
– Fred integrated another metadata extractor for a customer POC (on Flash files).

This means that we will plan some integration work to:

• package this inside DAM 2
• remove from DAM 2 the deprecated items.

  => need to integrate with DAM
  ==> avoid hard coded metadata schemas
  ==> provide default mapping
  ==> schedule work after DAM refactoring.

Deck.js

Laurent also worked on a template-rendering extension to add support for Deck.js PDF generation directly inside Nuxeo. The work includes:
– evolution of template rendering
– integration and dependency on Phantom.js.

The post [Nuxeo Tech Report] News from the Developer Front #3 appeared first on Nuxeo Blog.