Extracting Values from Element Attributes using Jsoup and a JavaScript Stage

While Fusion comes with built-in Jsoup selector functionality, it is limited in its extraction capability.  If you want to do something like extract attribute values -- in particular attribute values with special characters or empty spaces in the values, you'll need to do a custom JavaScript stage and implement the extraction there. 

 

To do this:

1) Create a custom JavaScript stage and order it directly after the Apache Tika Parser. 

2) Add your code.   For the purposes of this article, I've created the following example.  Depending on what you're trying to accomplish, your code may vary:

function(doc){
    
    var File = java.io.File;
var Iterator = java.util.Iterator;
var Jsoup = org.jsoup.Jsoup;
var Document = org.jsoup.nodes.Document;
var Element =  org.jsoup.nodes.Element;
var Elements = org.jsoup.select.Elements;

var content = doc.getFirstFieldValue("body");
var doc = org.jsoup.nodes.Document;
var e = java.lang.Exception;
var div = org.jsoup.nodes.Element;
var img = org.jsoup.nodes.Element;
var iter = java.util.Iterator;
var divs = org.jsoup.select.Elements;


  
   
   try {
             doc = Jsoup.parse(content);
             divs = doc.select("div");
             iter = divs.iterator();
            while (iter.hasNext()) {
                div = iter.next();
                if (div.attr("id").equals("featured-img")) {
                    break;
                }
            }

            if (div != null) {
                 img = div.child(0);
                logger.info("SRC: " + img.attr("src"));
                logger.info("ORIG FILE: " + img.attr("data-orig-file"));
                doc.addField("post_image", img.attr("src") + " | " + img.attr("data-orig-file"));
            } else {
                logger.warn("Div was null");
            }

        } catch ( e) {
           logger.error(e);
        }

    return doc;
}
          

 So let's go ahead and break down what is happening here:

1) Declare Java classes to be used. 

                       
var File = java.io.File;
var Iterator = java.util.Iterator;
var Jsoup = org.jsoup.Jsoup;
var Document = org.jsoup.nodes.Document;
var Element =  org.jsoup.nodes.Element;
var Elements = org.jsoup.select.Elements;

2) Next, declare our JavaScript variables to be used. Note that we assign the content variable to be the content pulled by the Apache Tika Parser

var content = doc.getFirstFieldValue("body");
var doc = org.jsoup.nodes.Document;
var e = java.lang.Exception;
var div = org.jsoup.nodes.Element;
var img = org.jsoup.nodes.Element;
var iter = java.util.Iterator;
var divs = org.jsoup.select.Elements;

3) Next, we pull the "div" elements out and look for one with an ID of "featured-image." Once we find it, we 'break' the iteration and move on. Note: I'm using this type of example to illustrate how to work with element attribute values that contain special characters or empty space. Jsoups selector syntax doesn't really play well with these types of characters.

 doc = Jsoup.parse(content); // parse the document
             divs = doc.select("div"); // select all the 'div' elements
             iter = divs.iterator(); // get an iterator for the list
            while (iter.hasNext()) { // iterate over the elements
                div = iter.next();
                if (div.attr("id").equals("featured-img")) { // if we find a match, assign and move on. 
                    break;
                }
            }

4) Finally, we set the values in the document.

   if (div != null) {
                 img = div.child(0); // get the image element
                logger.info("SRC: " + img.attr("src"));
                logger.info("ORIG FILE: " + img.attr("data-orig-file"));
                doc.addField("post_image", img.attr("src") + " | " + img.attr("data-orig-file")); // set the values in the PipelineDocument
            } else {
                logger.warn("Div was null");
            }

 

 

And that's all there is to it! Happy Extracting!

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk