Content Migration - How we got a project over a finish line 16.9 times faster

Last year we ended up migrating 38 web sites and major site sections (10,710 pages in total) in just over a week each. A Nation-Wide reputable vendor estimated each of them to take 3 to 6 months. How did we do it?

The short answer is this: we used Batch Loader. Easy enough. Am I simply comparing a manual import to the use of a tool? Nope. I'm not about to waste your time. After all, that vendor was also using batch loader.

Batch Loader Turbo Charged

When it comes to a mass-check in - batch loader is a nice and useful tool but when you're loading tenths of thousands of files from dozens of locations and when each file has unique derived values in its metadata - Batch Loader won't be of much help.

I guess that almost any enterprise-scale content migration will have you fall flat on your face if you're simply relying on Batch Loader to "magically" load your content into ECM.

So what is the quickest way to automate such a migration?

Its simple. The answer becomes obvious when you look at HOW the batch loader works. It processes a typical HDA file - one record at a time. It picks up a file from the location you specify in primaryFile field, sets metadata values to the ones you tell it to use and calls a Check In service. Again, it reads batch loader script one record at a time and checks in files - one by one.

What if we could create some cool batch loader script that will import all the files we want imported? All at once! Sure, that would be nice, but how do we go about creating one?

The Batch Builder utility that comes with Batch Loader is very limited. It builds a very simple files based on a content of a single directory lets you use file system data as meta. It won't let you pick up files from multiple locations or create complex meta values.

So, here's the biggie - to turbo-charge your content migration effort, you need may consider GENERATING your own batch loader scripts.

How to use Code Generation effectively

How do you go about generating it? For simple migration you can get away with using your editor's search and replace function on a comma-separated list of files

Let's say, your excel file has the following columns:

Content Id, Title, Author, Security Group, Account, Doc Type, Date, File Path

After you save it in a comma-separated (CSV) format, you'll end up with something like this:

A2561405, Migration Project Plan, Bill, Public, , abstract, 8/12/09 4:20 PM, C:/Migration/Project Plan v.3.4.doc

Now, you could use a RegEx like this to produce a batch loader script out of your CSV file:

Replace this:

^([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)$

( If you're new to regular expressions, this says:

- Begin at the start of a line
- Select every character until you see a comma - repeated 8 times
- You must be at the end of the line )

With this replacement string:

dDocName=$1\n dDocTitle=$2\n dDocAuthor=$3\n dSecurityGroup=$4\n dDocAccount=$5\n dDocType=$6\n dInDate=$7\n primaryFile=$8\n <<EOD>> \n

You may need to test the RegEx in your own editor as every one has a slightly different syntax.

After you run it - your comma-separated line will transform into an HDA entry that will look like this:

dDocName=A2561405
dDocTitle=Migration Project Plan
dDocAuthor=Bill
dSecurityGroup=Public
dDocAccount=
dDocType=abstract
dInDate=8/12/09 4:20 PM
primaryFile=C:/Migration/Project Plan v.3.4.doc
<<EOD>>

I hope, you get the idea.

How to scale it up

You can easily adapt this technique to any complexity. Just use Perl, Ruby or another scripting language of your choice to generate metadata values and the file names and locations.

Be sure to use subroutines, structure your code well and store it in source-control system. Code generation script can get quite complex quite quickly.

Important last minute tips

Today, there will be three:

- You'll very likely need to debug your code generation script and run your batch load file more then once so be sure to:

• Add a custom meta field or a special value like "batch_loader" for the dDocAuthor so you can find your files quickly and delete them when its time to start fresh
• Test on a small sub-set (under 200 items) so you don't have to wait for 8 hours for these 500 Gb to import

- Be sure to CLEAR the "Clean up files after successful check in" box. If you leave it checked - your source files WILL be deleted and you won't find them in a Recycle Bin!

- Be sure to MARK "Enable error file" box. This will create a detailed log file and a smaller batch loader script file for the files that didn't load. Absolutely essential!





That's all for now.

Happy Migrating!

How to build your own personal and portable testing and learning environment

I promised to walk you through the process of creating your own personal and portable testing and learning environment - the one you can load on your laptop so you won't be dependent on your sysadmin to give you permissions. You won't be afraid to break something and will feel free to experiment.

Having your own environment may also let you work from home - but don't mention my name when you talk to your boss :)

Let's get started.

In my courses I'm making an assumption that you're familiar with virtualization. In case you aren't - here is a 1 min introduction.

Virtual machine or VM is an instance of a real OS such as Windows or Linux (Guest OS) running on a special software you install on your machine. Now that hardly any computer has less then 1 Gb or RAM and most OS (except Vista, of course) take less then 512 Mb - you can run a second OS in your own virtual environment.

The benefits are multiple:

• Guest OS has absolutely no affect and may have no access to your host OS so even if you're running a virus in your guest OS - you are safe
• Moving VM on another machine is a simple file copy operation so if you have your VM on a portable drive - you can work on the same OS at home and at work.
  
• You don't need to pollute your software configuration when you can install and test one-offs on your virtual environment that can be reset to its original configuration in a matter of seconds

"OK, Show me how to build a virtual environment"

Here's how:

If you running Windows, begin by downloading and installing MS Virtual PC 2007. This is free download. On Linux - check out VMware products such as VMware Player or Server (currently free) or Workstation.

Here's a link to get MS VPC 2007

Download and unzip a sample VM. The one I'm recommending comes with Windows Server and SQL Server 2005.

The link is here

In VPC 2007 go File - New VM Wizard -> Use Default Settings to Create a VM.

Once VM settings are open, under hard drive select the location of your newly download SQL Server 2005 machine VHD file.

Give it about a 512 Mb or RAM and hit Start.

Voila! You're now ready to begin Oracle Content Server installation.


Please follow the instructions in my YouTube Video to install your Content Server

Since you are going to be using MS SQL Server, check out my article for JDBC Driver instructions

In the next 15 min you should have your new portable installation of Oracle Content Server up and running. You can put your (updated) VHD file on a portable drive and take with you wherever you go!

Are you getting ripped off by your Oracle ECM consultant?

Its absolutely maddening! A couple of days ago I've got a call from Alex. He was too busy to help out a client and wondered if I could help them out. I asked what do they need done. Guess what? Yet another new Oracle ECM client was badly burnt by a large consulting shop.

First they paid $250/hr plus travel and accommodation for a consultant to come on site for two weeks and "design an implementation approach". What they ended up getting for over $25,000.00 is a copy of their own technical architecture document with a few pages of Oracle "Planning and Implementation Guide" plugged in. That's all! Like that wasn't enough?

The client proceeded anyways. They invested in the Content Server licenses and needed ECM up... Another month and a 25% of their budget later they had a few requirements workshops done and a requirements gathering template partially filled in. They've seen a PM, BA, an ECM consultant and an architect. No results! Isn't that NUTS?

That amount was sufficient to have their environments rolled out and users trained in ECM but all they got is a couple of useless documents. What a rip-off!

I understand that replying to an RFP costs money but why the client has to pay for it?

If any one of you tells me another story like this - I'll be banging my head against the wall and screaming until my neighbour comes back from their night shift and stops me! I cannot take it anymore! Can't you guys see who you hiring?

OK, I understand that frustration doesn't help much so let me give you a few red flags look for

• Forced methodology. When consultant "recommends" very insistently that you have them produce a document or other "artefact" because its required by their methodology - be sure to question that and get your own independent assessment of what kind of return will you'd be getting on this investment.
  
  

• Withholding knowledge. Unless you plan on retaining the consulting shop indefinitely, put very specific controls in place regarding the knowledge transfer. Be sure that consultants are ready and willing to explain every step they take and help your full time staff to follow by their footsteps. Verbal promises on behalf of the sales team are usually not enough to get that in place.
  
  

• Unnecessary consultants. Before authorizing a team of consultants to start working on your project - are you sure you understand what every one of them is doing? Are you sure you will be getting value for what you pay for EACH ONE of them?
  
  

• Inflated estimates. This one hardly needs any explanation. As long as you don't have in-house Oracle ECM expertise - you are completely at they mercy as far as the estimates are concerned. They are free to double and triple the time things actually take and you won't even know when to object.
  

• Weak or no guarantee. Typical 30 days guarantee when consultants fix the bugs you find after the project is complete - is not much help:

  • Does it protect you from the project taking five times longer then they promised?
    

  • What about the one that delivered five times less then you expected where everything over and above the out-of-the-box Content Server deployment is a "Change Request" and costs extra?
    

  • Also - do you seriously expect them to fix bugs FOR FREE at the same pace they did when they were paid? Come on! Those developers were reassigned to another project a week before your last payment came out!
    

Hope those will make you a little more prepared to deal with not-so-honest consultants or minimize your losses if you already have one of these by your side.

Can they still cheat you?

Well, now that you will be noticing the "tricks of the trade" it will be harder for them to do but many will continue to get away until you have an in house Oracle ECM expertise ... and - despite of what they tell you

- you actually can get good working knowledge of Oracle ECM in just a few days and

- NO - it won't require sending your people for an expensive several day course.

For starters, why don't you get a "sandbox" copy of Oracle ECM on your laptop? You can get an easy, step by step instructions here and simply play around with it. Online help, combined with information from this blog should get you pretty far, pretty fast.

I'm working a complete set of FREE or very LOW COST training materials to get you the knowledge you need as quick and easy as it is humanly possible - you can check out my video courses at stellentexperts.com. And, of course, keep an eye on this blog.

That's all for now,

Best,
Dmitri

Fastest way to understand security model I know about

At last! After years of struggling to explain Content Server security model to my clients - I finally got it!

A while back when I was first learning it myself - it took me about an hour to understand how Role-based component alone is used. But when I tried to add on accounts... Later in that day there were five of us desperately trying to cut through the mess of multiple dimensions and confusing terminology. It took us several hours of screaming around a white board just to begin seeing some light at the end of the tunnel...

It always bothered me greatly to look at my customers suffer through trying to get their head around it. Many just gave up and were forced to rely on consultants for explanation... Not any more!

So here it comes

I finally realized that most humans are not good at piling on multiple dimensions ... and I see no benefit of looking at all the components of security model at once! When clients were introduced only one component of the model at a time and asked to keep them separate in their heads (and their diagrams) - the confusion quickly subsided or didn't appear in the first place!

Check out my FREE video tutorial at http://youtube.com/stellentexperts

Gain Complete Working Knowledge Of Content Server In 5 Days Or Less In Just Minutes A Day - 100% GUARANTEED

Content Server Running Start Course is now available!

I did my best to put everything you'll need to stand comfortably on your own two feet, so you won't have to spend a single penny on consulting or other Content Server education... and this is the easiest course to follow that I know about...

Seasoned Professionals - don't be so quick to dismiss it! Are you sure you're crystal clear on all the topics I cover like building a portable virtual testing environment and my own approach to security model?

Check it out at http://www.stellentexperts.com/se/running_start.htm

Are you crystal clear on Stellent security model

Last week I was doing one of my presentations on Stellent security model. Its stunning how after all these years this of all Stellent aspects, still manages to confuse even most experienced admins. ARE YOU CRYSTAL CLEAR ON STELLENT SECURITY MODEL? If not – here's your magic bullet – it requires a paradigm shift.

Start with the basic terminology. You will need to understand what Stellent means by such commonly used terms such as "group" or "account". Believe me – its' not what everyone's thinking!

When you get comfortable with the glossary – draw a diagram how all of this fits together. Drive home how content is assigned into groups, how group access is controlled by roles and how account hierarchy works.

Once you get that down – check how Folders are used to impose virtual hierarchy on otherwise flat repository.

Check out my presentation below. It covers the same ground in a little bit more details and may become a good road map for you…. Good luck

Oracle Stellent
SECURITY model

Lessons learned

Dmitri Khanine

dk at stellentexperts dot com

Oracle Stellent Security

• Powerful but can be confusing

• Requires careful planning

• Professional help is not always an answer

– Why? Explained later in this presentation….

Security Model Overview

• Any security model made of:

– Authentication

– Authorization

• Authentication

– Stellent

– Windows

– Active Directory

– LDAP

– Custom

• Authorization

– Groups

– Roles

– Accounts

• Authentication and Authorization combo:
(LDAP, Active Directory, Custom)

– Allows mapping of existing org structure to Stellent security principals

– Stops the agony of maintaining permissions of the same user in multiple places

– Covered in detail later in this presentation

Security planning

• Long ranging implications of overly complex and under-designed models

– Users will be able to do more then it is mandated or even safe for them to do

– Access may be denied in unpredictable situations

– Hard to debug user access issues

– New content will follow the model and increase the cost of correcting the model

On Content Revisions

• Major benefit of content management…

• Each revision can have different metadata

– To update security on a content item all revisions need to be updated

– Even if you don’t add new content – changing of security gets more expensive over time

Security planning

• Long ranging implications of overly complex and under-designed models

• Easier to start with a good model then change later

BUT

• Security model can be changed at any time

… How to plan a security model – later in this presentation

Basic terminology:Major source of confusion

• Groups –

– Windows: a set of users

– Stellent: a piece of metadata

• Accounts –

– Windows: user record

– Stellent: a piece of metadata

Security at a glance

• Groups and accounts used together

Why is it so confusing?

• Robust, time tested system

• Hundreds of customers worldwide

• Is it just terminology?

Paradigm shift

• Single flat repository instead of conventional taxonomy

– No permission inheritance in the base system

– Optional Folders components required for some of support hierarchies BUT

– Folders are no longer synchronized with web site navigation (as of v. 7.x)

Groups

• A “tag” on a content item

• Helps to group items with same security level

• Used verbatim

• Limited to 30 characters

• Access to groups specified by user role membership

Roles

• Specifies user access level to various groups

• User can belong to multiple roles

Accounts

– Another “tag” on a content item

BUT

– Supports hierarchy

– User access to accounts specified directly

• Accounts support hierarchy, based on a “Starts with” substring rule

• RW on ca/on will give Read and Write access to

• ca/on/montreal

• ca/on/toronto

• ca/on/Ottawa

– Slashes are optional

• RW on ca,on will give Read and Write access to

• ca,on.montreal

• ca,on-toronto

• Ca,on=>ottawa

– Length limited to 35 characters

– Can be created on the fly…

Ignore these and get in trouble:

• Account text box lets people create ad-hoc accounts that others can not access

How I plan Stellent security

• Name all types of people who will at any time access the system

• Will they be authenticated? How?

• What will they do on the system?

Case study

• Model features after security redesign:

– Roles follows user’ job function, not organizational unit

– Groups describe access level

– Accounts segment content by organizational unit and web site section

Oracle Stellent Security:

• Requires careful planning

– Authentication

– Authorization

- Groups, Roles, Accounts

BUT

• Bad model can be corrected at any time

• Requires paradigm shift in thinking and can be confusing

• Professional help is not always an answer

– Must have experience implementing Stellent security models

Must be able to explain Stellent Security concepts without mystifying them

• Questions?

How to remove account text field

Content Server 7.x as well as 10G includes an editable text box next to Account field on content check in form. Allowing users to type in arbitrary accounts increases the risk of newly checked in content becoming inaccessible to others and has a potential of causing performance problems due to proliferation of account-related directories in weblayout. Many people wonder if they can remove it so users can only pick from the list of pre-defined accounts.

Here is how to accomplish that in just a few minutes without resorting to custom component development.

1. Start Configuration Manager

2. On the Tables Tab go Add table

3. Select DocumentAccounts table, Click OK

4. On the Views Tab go Add…

4.1 Pick DocumentAccounts as your table and select dDocAccount field so it will be shown

4.2 On Security Tab uncheck the Publish view data and select “Use standard document security” as shown below

4.3 Hit OK to close Edit View dialog

5. On Information Fields tab hit Add…

5. 1 Type field caption such as “SecurityAccount” , pick desired field order

5.2 Check the Enable Option List checkbox as shown below and click Configure button besides it

5.3 Select the Use View option as shown below. Select the view you’ve created on step 4 as shown below

5.4 Click OK to close the Configure Option List dialog

5.5 Click OK to close Edit Custom Info Field dialog

6. Click Update Database Design button (right side on the Information Fields tab on Configuration manager)

7. Select the Rules tab and click Add…

7.1 Type in a name for the new Global Rule you will be creating

7.2 On the Edit Rule dialog make sure that “Is global rule with priority” box is checked

7.3 Select Fields tab (see below)

7.3.1 Click Add to add a new filed

7.3.2 Select the filed you’ve created on step 5

7.3.3 Change Type to required or leave as Edit depending on your preference

7.3.4 Add dDocAccount field as you did on step 7.3.1 but this time mark is Hidden

7.3.5 Make sure that “Is derived field” box is checked and click “Edit…” as shown below

7.3.6 Switch to Custom tab on the Script Properties dialog as shown below

7.3.7 Check “Custom“ check box and paste the following text into it:

<$dprDerivedValue=getFieldViewValue("xSecurityAccount",#active.xSecurityAccount,"dDocAccount")$>

7.3.8 Click OK to close Script Properties dialog.

7.3.9 Click OK to close Edit Rule Field dialog

7.4 Click OK to close Edit Rule dialog

8. Select Options -> Publish Schema and Options -> Publish Schema Base so that your changes are visible on the check in form

WARNING : The global rule created on step 7 will hide the built in Account filed on content information, search and update screens to above steps will only be sufficient if you perform them on a new instance of content server. If you are working on an instance with existing content – follow the steps below:

9. Under Rules tab of Configuration Manager select the rule you’ve added on step 7 and hit Edit…

9.1 Select “Use rule activation condition” as shown below

9.2 Click Edit… and Add… to add a rule condition as shown below.

This will ensure that the original Account field is still visible and none your users are affected when they perform and update. You might want to add another rule to hide your newly created filed on search and content information pages…