 | Level: Intermediate Jimmy Thrasher (jjthrash@us.ibm.com), Software engineer , Pervasive Computing at the IBM Research Triangle Park facility in Raleigh, North Carolina
11 Sep 2002 This article discusses the Transcoding Technology feature of WebSphere Everyplace Access, which enables you to make portal information available to multiple types of devices. It includes an overview of the Transcoding Technology architecture.
Introduction WebSphere® EveryplaceTM Access (hereafter
called Everyplace Access) lets developers and administrators build
wireless applications quickly and effectively. Transcoding Technology,
a component of WebSphere Everyplace Access, enables one to make
information on the Web available to multiple types of devices (for
example, PDAs, cell phones, etc.). Simply, Transcoding Technology
converts content from one form to another. It includes conversions
from one markup language to another and content annotation of HTML.
It is also pluggable and includes a powerful architecture for developing
specialized transformations. This article discusses the Transcoding Technology
feature, and specifically focuses on what this tool provides and
how developers can use it effectively in a pervasive portal. An
overview of the Transcoding Technology architecture is also provided,
along with a description of the various tasks that developers can
accomplish with Transcoding Technology in Everyplace Access. This
article assumes some knowledge of how an enterprise portal works.
For introductory material, please refer to the Related
resources section. A stylesheet example is also provided with this
article. See the Download section below.
What is transcoding? Transcoding is the process of dynamically mutating
and customizing a document based on the characteristics of the requesting
device and on the user's preferences. Transcoding Technology is
the component of WebSphere Everyplace Access that provides portlet
developers and portal administrators with transcoding capability.
The Transcoding Technology component provides two essential functions:
deck fragmentation and annotation, which the Web clipping portlet
uses. This article will discuss both deck fragmentation and annotation,
but will not discuss the Web clipping portlet. For details on the
Web clipping portlet, refer to Scott DeWitt's article, Basic
Web Clipping Using WebSphere Portal Version 4.1, which was published
in June 2002 on the WebSphere Developer Domain. Many Transcoding Technology functions will be
discussed in this article.
Architecture To get the most out of this article, I recommend
that you become familiar with the basic WebSphere Portal (hereafter
called Portal) architecture by referencing the various items listed
in the Related resources section below. WebSphere Portal invokes Transcoding Technology
at two different levels: the individual portlet level and the full
page level. The flow of a request is illustrated in Figure 1:
Figure 1. WebSphere Portal and Transcoding Technology
request flow
The above diagram can be understood as follows:
- The client device makes a request to the portal, which the Portal
Filter receives.
- The Portal Filter calls WebSphere Portal to receive its contents,
which then calls the Aggregator to aggregate the page.
- The Aggregator selects the portlets based on a combination of
values. If a portlet is configured to be transcoded, and it provides
a markup that Transcoding Technology can use, then it will be
selected. For example, if the portlet provides HTML and the client
needs WML, then the Aggregator will select the portlet because
it is transcodable. If a portlet is configured to use Transcoding
Technology, then the Portlet Filter will be placed in front of
the portlet.
- The Portlet Filter calls the portlet to receive its contents.
- The portlet returns its contents.
- The Portlet Filter sends the portlet's contents to Transcoding
Technology for processing, associating the portlet's contents
with a default URI unless otherwise specified (see the Portlet-Level
Transcoding section). Note that the portlet thinks it is talking
directly to the Aggregator, so no special programming style is
needed to write a portlet that is filtered.
- Transcoding Technology returns the processed contents to the
Portlet Filter.
- The Portlet Filter returns the processed contents as if it were
a proper portlet returning contents normally.
- The Aggregator, having finished aggregating the page, returns
the page.
- The Portal Filter sends the portal's aggregated contents to
Transcoding Technology for processing.
- Transcoding Technology returns the processed contents.
- The Portal Filter returns the processed contents to the client.
 |
General transcoding concepts Certain aspects of Transcoding Technology apply
to both portlet-level transcoding and portal-level transcoding,
so we will cover these aspects first: Device profiles WebSphere Portal has a simple and effective method
for determining device characteristics. A regular expression, meant
to match a particular User-Agent string, is mapped to a client,
which has an associated aggregator and set of capabilities. This
associated aggregator specifies a particular markup. For example,
the HTML aggregator specifies text/html as its markup.
Transcoding Technology has a separate device profile system with
a separate set of device profiles that is tied into the portal as
follows:
- The specific client is determined by the User-Agent field in
the HTTP request headers. This client has a manufacturer property,
and may also have model and version properties. An example of
properties for Microsoft® Internet ExplorerTM would be
Microsoft, Internet Explorer, and 6.0.
- The manufacturer, model, and version are combined to form a
unique device profile name, which is passed into Transcoding Technology.
For example,
Microsoft, Internet Explorer,
and 6.0 would be combined to form Microsoft!Internet+Explorer!6.0.
- Transcoding Technology matches this unique name with a predefined
device profile.
When clients are created and modified in Portal,
they are automatically propagated into Transcoding Technology's
device profiles; this means that the user does not have to worry
about keeping them synchronized. However, the Transcoding Technology
profiles have some properties that aren't shared by Portal, since
they are transcoding-specific. Some of the more important ones include:
- WMLMaximumDeckSize The maximum deck size for a WML document.
Fragments will be this size or smaller.
- HTMLMaximumPageSize The maximum page size for an HTML
document. Note that this is for i-mode or cHTML devices.
- TextLinksPreferredToImages This instructs Transcoding
Technology to convert an embedded image into a link that points
to that image in order to reduce the amount of valuable space
that an image takes up on a device.
- supportedImages A list of image types that the device
supports.
- imageScalingValue A fraction that specifies how much
to scale images.
- colorSupported Indicates whether the device supports
color.
- colorDesired Indicates whether the device wants color.
- screenCapability Indicates a screen resolution of high,
medium, or low.
- colorDepth The bit depth of the number of colors that
are supported by the device.
To change one of these settings, you must use
the XML configuration tools. XML configuration The Transcoding Technology XML configuration
tools included with Everyplace Access is very similar to the XMLConfig
utility that ships with WebSphere Application Server. There are
two main batch files, ExportResources.bat and ImportResources.bat;
both are located in the Transcoding Technology installation directory
(on Windows, this would be c:\Program Files\IBMTrans,
on Linux, /opt/IBMTrans, and on AIX, /usr/IBMTrans). To make any changes, you must first run ExportResources.bat.
The default options (i.e., no parameters) will cause all of the
configuration to be dumped into the WTPResources.xml
file. Because the file is in XML format, you can use any text editor
to edit it. For example, let's say that you have an AlcatelTM WAP
phone, and want to increase the WMLMaximumDeckSize value to 5K (please
note that here, I am simply making up a limitation for illustration
purposes). According to the Manage Clients administration portlet
in Everyplace Access, the manufacturer value for this client is
Alcatel. However, this client does not have a model
or version property, so the Device Profile that Transcoding Technology
will use is simply named Alcatel. Therefore, after you open the WTPResources.xml
file and search for Alcatel, you will find a Device
element with a SelectorName child that contains Alcatel.
It should look something like this: <Device>
<SelectorName>Alcatel</SelectorName>
<UserAgent>(User_Agent=*Alcatel*)</UserAgent>
<DesiredMimeType order="0">text/vnd.wap.wml<DesiredMimeType>
<SupportsJavaApplets>false</SupportsJavaApplets>
<SupportsJavaScripts>false</SupportsJavaScripts>
<SupportsFrames>false</SupportsFrames>
<CreateCHTML>false</CreateCHTML>
<IsPortalClient>true</IsPortalClient>
<PortalOrdinal>40</PortalClient>
<PortalParent>WML-Device</PortalParent>
<Enable>true</Enable>
</Device> |
This Device element is the entry
that corresponds to the Alcatel client in Portal. Unfortunately,
we can't find a WMLMaximumDeckSize entry in this device
profile, so we must look for the PortalParent child
element. This is the device from the Alcatel entry. We see that
it is WML-Device, and after looking at the WML-Device
entry, we see that the default WMLMaximumDeckSize is
2880 bytes. To change this for Alcatel phones, we need to add the
following to the Alcatel Device element: <Keys>
<Key Name="WMLMaximumDeckSize">5120</Key>
</Keys> |
Then, save the file and run ImportResources.bat,
which defaults to using the WTPResources.xml file.
The decks will now be fragmented to 5K instead of to 2880 bytes
for Alcatel phones. Transcoders Transcoding Technology's main function is built
around a set of plugins called transcoders. This includes annotation,
markup language conversion, and so forth. It is designed to let
others write plugins for Transcoding Technology. I will briefly
cover how to use the XML Configuration tool to register a transcoder
that you've already built. For more details on writing transcoders,
refer to the Related resources section
below. Let's say that you have a transcoder contained
in myUsefulTranscoder.jar. You will need to create
an XML file to register it: <Resources>
<Plugin>
<Location>c:/mydirectory/myUsefulTranscoder.jar</Location>
<Force>true</Force><!-- optional parameter -->
<Enable>true</Enable><!-- optional parameter -->
<Plugin>
</Resources> |
Now, run ImportResources using the
following command: ImportResources -File myFile.xml.
The Location element in the XML file will tell the XML Configuration
tool where to find the transcoder.
Portlet-level transcoding Portlet-level transcoding is used for content
transformations at the portlet level. This includes markup conversions,
annotations, stylesheet applications, and any other conversion that
affects an individual portlet. As was mentioned in step 6 of the
Architecture section above, the portlet's
contents are associated with a URI when passed through Transcoding
Technology. By default, this URI is http://<hostname>/%%WPS/Portlet.
However, the URI can be modified by setting the PortletURI
parameter in the portlet configuration. This URI can be used for
transcoders, and along with other resources that use it. Configuring a portlet to use transcoding In the most basic case, to configure a portlet
to use Transcoding Technology, you can do one of the following:
- Add a parameter to the portlet descriptor,
portlet.xml,
as a <config-param> in the concrete
portlet definition, or
- Add a parameter via the manage portlets administration portlet
using the Modify Parameters option.
In general, you will most likely have to decide
between two options:
- You want the portlet to be available in all markups. To configure
this, add the parameter
FilterChain=Transcoding to
the portlet, or
- You only want the portlet to be transcoded to some markups.
For example, if the portlet provides only HTML, and you want it
to be transcoded to WML, but not cHTML, then do the following:
Open the file <AppServer>/lib/app/config/services/PortletFilterService.properties
in a text editor. You will see the following entry:
filtername1=Transcoding
Transcoding.classname =
com.ibm.transform.wps.portletfilter.TranscodingPortletFilter
Transcoding.transcodeMarkup.1 = html->wml
Transcoding.transcodeMarkup.2 = html->chtml
Transcoding.transcodeMarkup.3 = html->vxml
|
This is the default Portlet Filter. It is configured to
support conversions from HTML to WML, HTML to cHTML (compact HTML
for I-mode phones in Japan), and HTML to VXML (Voice XML for voice
servers).
Add the following entry:
filtername2 = wmlTranscoding
wmlTranscoding.classname =
com.ibm.transform.wps.portletfilter.TranscodingPortletFilter
wmlTranscoding.transcodeMarkup.1 = html->wml
|
This creates a Portlet Filter that will only transcode
from HTML to WML and HTML to HTML. When you configure your portlet,
use FilterChain=wmlTranscoding instead of FilterChain=Transcoding.
The portlet will only be selected if the client device wants HTML
or WML, assuming that the portlet provides HTML.
Use this same technique every time you want to
limit when a portlet gets selected. For more complicated cases,
such as when you want it to be selected for WML and cHTML, but not
for VXML, then you will have to create a wmlChtmlTranscoding
filter with two transcodeMarkup entries, one each for
WML and cHTML.
Annotation The Transcoding Technology Annotation feature
lets you modify the structure of an HTML document via an XML-based
Annotation Language. The following is a quick overview of how it
works and what you can do with it. See the Related
resources section for links to information on annotators and
how to build them. How annotation works Annotation works on a Document Object Model (DOM)
version of a parsed HTML document by applying annotations to DOM
elements that are selected by XPath statements. For example, the
statement: <description take-effect="before" target="/descendant::TABLE[2]">
<remove />
</description> |
will apply the remove annotation
immediately before the second table in the document. The annotation
engine augments the DOM by adding annotation nodes where the XPath
statements match. After the DOM is augmented with annotation statements,
the Annotation Engine makes another pass through the DOM, and processes
the annotations. In the <keep/> and <remove/>
cases, the annotation engine turns element removal off and on, respectively,
meaning that the engine will throw out all elements that it finds
after a <remove/> annotation until it hits a
<keep/> annotation. Other annotations include markup replacement,
markup insertion, and splitpoints. Here's a quick summary of the more useful annotations:
- <remove> Turn on the remove state.
- <keep> Turn off the remove state.
- <table> Use for annotating tables.
- <column> Use inside the table annotator to select
an entire column.
- <row> Use to select an entire row.
- <field> Use to modify fields in a form.
- <option> Used inside the field annotator to specify
a selectable option.
- <insertattribute> Insert an attribute in an HTML
tag.
- <inserthtml> Insert an attribute in an HTML tag.
- <replace> Use to replace the selected element in
your document with the specified content.
- <replacewithhtml> Use to replace the selected element
with HTML markup.
- <setpreference> Use to set a preference that Transcoding
Technology will use when transcoding.
- <splitpoint> Specify in an HTML document a fragmentation
hint for fragmentation.
The various uses of annotation Often, it would seem easier and faster to write
the portlet source using a JSP file and conditional statements,
instead of writing an annotation that needs to be applied. Therefore,
understanding where and when annotation can be used is very important
in using annotation effectively.
- Annotation is useful if the person generating the markup for
the portlet is not comfortable with writing Java code to create
or modify the structure of a document.
- Annotation is useful for processing content that is not directly
generated by the portlet itself, or content that you do not have
control over.
- Annotation is useful for adding markup by using XPath statements.
- Annotation is useful for device-specific annotations.
Configuring a portlet to use annotation To apply annotation to a portlet, you must first
specify that the portlet is to be transcoded (see the Portlet-level
transcoding section above). Then, you have the following options:
- Specify the annotation file via a URL in the portlet's configuration.
You can do this by either adding a
config-param to
the portlet.xml file, or by using the manage portlets
administration portlet to add the parameter AnnotationFile=<url>,
where <url> is the URL of the annotator.
This can be any URL that is accessible by the server machine.
It is important to note that in the case of a file URL (for example,
file:c:/path/file.ann), it must be on the server
machine itself.
- Register the annotation file into Transcoding Technology itself,
using the XML Configuration tools. Now, in order for this particular
scenario to work, you will need to use the
PortletURI
parameter mentioned above so that the registered annotator will
know which portlet to annotate, so that it doesn't annotate all
of the portlets indiscriminately.
- Use internal annotation. This lets you embed annotation statements
such as the following into your document:
<p>Here is some HTML content that might have been removed by a prior <remove/> tag</p>
<!--
<?xml version="1.0" encoding="ISO-8859-1"?>
<annot version="1.0">
<keep/>
</annot>
-->
<p>Here is some more HTML that will get kept, since we found a <keep/> tag</p> |
Notice the lack of XPath entries. When you embed
the statements directly into your HTML source, the annotation engine
already knows where to annotate, so they become unnecessary. Annotation for multi-page portlets If your portlet has multiple pages and you want
to apply annotation to them, then you are limited to either using
internal annotation, or ensuring that each page has the same structure
(this way, you can apply the same annotator to each page). It is
important to keep the same structure because the XPath statements
that make up an annotator rely heavily on the structure of the document;
if the structure is not the same, then the annotators will not be
applied as expected.
Stylesheets One popular way to convert from XML to any markup
language is by using XSL stylesheets. Tools such as Xalan use stylesheets
to process the XML content according to the included instructions
in the stylesheets. Transcoding Technology lets you apply stylesheets
to portlets that generate XML content. There are some qualities of Transcoding Technology's
XSL handling that make it very useful for multi-device scenarios.
For example, it provides device properties to the XSLT engine that
let you parameterize the stylesheet based on device characteristics,
such as screen resolution, whether color is supported, and so forth.
This lets you customize your transformation, which becomes especially
useful if you want to create only one stylesheet to generate HTML
from your XML, but you would also like to support WML. In this case,
you could parameterize your stylesheet to generate a smaller HTML
document for a WML device, which Transcoding Technology would then
transcode to WML (see the article by Alan Booth and Kathryn Britton,
Spinning
your XML for screens of all sizes, published on IBM developerWorks). Configuring WebSphere Portal to support the use of stylesheets Before configuring the portlet, you must configure
the portal. However, you only need to do this once. Complete the
following:
- Open the file
<AppServer>/lib/app/config/services/PortletFilterService.properties
in a text editor.
- Find the following entries:
Transcoding.transcodeMarkup.1 = html->wml
Transcoding.transcodeMarkup.2 = html->chtml
Transcoding.transcodeMarkup.3 = html->vxml
|
- Add the following entries:
Transcoding.transcodeMarkup.4 = xml->html
Transcoding.transcodeMarkup.5 = xml->wml |
- Save and close the file.
- Log on to your portal as the Administrator, and select Portal
Administration => Portal Settings => Manage Markups.
- Click Add new markup.
- Enter
xml, text/xml, and UTF-8
for the markup name, MIME type, and default character set, respectively.
- Restart the portal.
Configuring a portlet to use a stylesheet
- When creating the
portlet.xml for your XML portlet,
specify the following in the <supports>
element:
<supports>
<markup name="xml">
<view output="fragment"/>
</markup>
</supports>
|
- After installing your portlet, select Portal Administration
=> Portlets => Manage Portlets.
- Select your portlet from the list, then click Modify Parameters.
- Add
FilterChain=Transcoding and StylesheetFile=<url_to_file>.
Make sure that the URL you specify for StylesheetFile
is accessible by WebSphere Portal itself, since Portal will be
prompting for the file. If the file is on the same machine that
Portal is on, then you can use a file URL, for example, file:c:/mydirectory/myfile.xsl.
A quick example RSS (Rich Site Summary) is a common format for
syndicated news feeds, Web logs, etc. It is XML, and therefore also
a prime target for an example in this kind of paper. I won't include
the XSL and XML details here; see the Related
resources section for the files I used. I used ServletInvokerPortlet that
is included in Everyplace Access, and packaged it as above, specifying
that it provides XML. Then, I created a JSP file to get the contents
of a URL and spit them out. This URL, http://www.xmlhack.com/rsscat.php,
is an RSS feed for XMLHack; it is a popular news site for XML information.
I set up the portlet to use this JSP source. As a result, I now
have a portlet that spits out RSS. Then, I wrote an XSL stylesheet to convert the
RSS into meaningful HTML. I used the parameterization feature to
key off the textLinksPreferredToImages device property.
This way, if a device doesn't prefer images, it will get text for
the title instead of the image. Because this XSL can be applied to more than just
my specific XML page, I decided to register it with Transcoding
Technology instead of using the StylesheetFile property.
When a stylesheet is registered with Transcoding Technology, it
can be applied to every XML that specifies a certain DTD, or that
matches a particular URL pattern. To do this, I first called the
ExportResources batch file to export all of the preferences.
I did this to obtain the ServerModel value for this
installation of Transcoding Technology, as well as for the Version.
Then, I stripped away unnecessary items until I got the following
XML file: <?xml version="1.0" encoding="UTF-8"?>
<Resources ServerModel="testOne" Version="magic411srv">
</Resources> |
Then, I added information for my stylesheet, and
saved the file as myStylesheetResource.xml: <?xml version="1.0 encoding="UTF-8"?>
<Resources ServerModel="testOne" Version="magic411srv">
<Stylesheet>
<SelectorName>RSS to HTML</SelectorName>
<Name>RSS to HTML</Name>
<Description>Converts an RSS feed into HTML</Description>
<OutputMimeType Enable="true" order="0">text/vnd.wap.wml</OutputMimeType>
<OutputMimeType Enable="true" order="1">text/html</OutputMimeType>
<Location>file:/C:/rss2html.xsl</Location>
<InputDTD>http://my.netscape.com/publish/formats/rss-0.91.dtd</InputDTD>
<Enable>true</Enable>
</Stylesheet>
</Resources> |
Please note that what is specified in the Location
element is on the server machine, where Transcoding Technology is
installed. I then ran the ImportResources.bat -File
myStylesheetResource.xml command to register it. After restarting
WebSphere Portal, the Web browser displays the following:
Figure 2. Output to the Web browser
Web clipping portlets Web clipping portlets let a portal administrator
create a portlet that consumes existing content from a Web page
and keeps only what is necessary. The process to create a Web clipper
is very simple. To see documentation on Web clipping, see the WebSphere
Portal InfoCenter's Web clipping page. Also refer to Scott DeWitt's
article, Basic
Web Clipping Using WebSphere Portal Version 4.1.
Portal-level transcoding Portal-level transcoding is used modify an entire
page. By design, the entire page is usually in the correct output
markup language by this point, so no markup conversions should take
place at this level. The main purpose for portal-level transcoding
is to provide deck fragmentation. Deck fragmentation Many wireless devices, especially cell phones,
can support only a limited page (or deck, as it is called in WML)
size, and if they receive a page that is too large, they will give
an error, or possibly even crash. After the portal page has been
assembled, Transcoding Technology will split the page into the correct
size fragments and serve the first fragment, caching the rest to
be served when the device requests them. This brings up the question,
what is the correct size for fragments? Transcoding Technology lets
you to modify these values to best suit your devices. See the example
in the Device profiles section. Fragmentation can sometimes lead to strange splits
in the document. To address this, Transcoding Technology lets you
insert fragmentation hints, so that you can indicate the appropriate
places to fragment the document. If you are generating the WML or
cHTML source yourself, then insert the following comment in your
document as a fragmentation hint:
Otherwise, if you are editing HTML that may become
transcoded into WML, then you can use the <splitpoint/>
annotation, either externally using XPath statements or internally.
See the Annotation section for details
on how to carry out either task. Make sure to avoid whitespace inside
the comment when you add this to your document.
Conclusion
Transcoding Technology is an excellent tool for
developers or administrators interested in making a pervasive portal
even more pervasive. Now that you have a basic understanding of
how Transcoding Technology works and how to develop your own transcoders,
you can take advantage of this function to build more powerful portals.
Download | Name | Size | Download method |
|---|
| stylesheetExample.zip | 2 KB | FTP|HTTP |
Resources
About the author | | | Jimmy Thrasher is a software engineer for Pervasive Computing at the IBM Research Triangle Park facility in Raleigh, North Carolina. You can reach Jimmy at jjthrash@us.ibm.com. |
Rate this page
| |
