Gravity Labs

Goose - Article Extractor

Extracts the main body, meta data and most probable image candidate from any news article or article type web page.

Overview

The extraction goal is to try and get the purest extraction from the beginning of the article for servicing Flipboard/Pulse type applications that need to show the first snippet of a web article along with an image.

Goose will try to extract the following information:

  • Main text of an article
  • Main image of article
  • Any Youtube/Vimeo movies embedded in article
  • Meta Description
  • Meta tags
  • Publish Date

This video introduces you to how we filter and extract the content used to build the interest graph. The Demo allows you to extract content of an article URL of your choosing.

Check out the Demo

Demo

Provide Goose with an article URL: identify the main image

Results

Title

Canonical URL

Meta Keywords

None

Meta Description

None

Article Text

  

Description

Goose was open sourced by Gravity.com in 2011 and is available on GitHub

Developer:

Contributors:

The GitHub wiki (https://github.com/GravityLabs/goose/wiki) has the full details on how to use Goose.

Licensing

Goose is available for free and released by Gravity.com under the Apache 2.0 license. See the LICENSE file for all the details.

If you find Goose useful or have issues please drop me a line, I'd love to hear how you're using it or what features should be improved

Take it for a spin

To use goose from the command line:

cd into the goose directory
mvn compile
MAVEN_OPTS="-Xms256m -Xmx2000m"; mvn exec:java -Dexec.mainClass=com.gravity.goose.TalkToMeGoose -Dexec.args="http://techcrunch.com/2011/05/13/native-apps-or-web-apps-particle-code-wants-you-to-do-both/" -e -q > ~/Desktop/gooseresult.txt

Regarding the port from JAVA to Scala

Goose was originally written in Java but was converted to a Scala project in August of 2011.

Here are some of the reasons for the port to Scala:

  • Gravity has moved more towards Scala development internally so maintenance started to become an issue
  • There wasn't enough contribution to warrant keeping it in Java
  • The packages were all namespaced under a person's name and not the company's name
  • Scala is more fun

Issues

It was a pretty fast Java to Scala port so lots of the nicities of the Scala language aren't in the codebase yet, but those will come over the coming months as we re-write alot of the internal methods to be more Scala-esque. We made sure it was still nice and operable from Java as well so if you're using Goose from java you still should be able to use it with a few changes to the method signatures.