Wrench value from your Creo Investments Home Measure the benefits Simple Automation can bring Why
Automate?
A personal way to do PDM Peer to Peer
PDM
3D Drawings Model Based
Design
Smooth out unwanted sharp edges Other
Articles
www.proetoolbox.co.uk - Simple Automation made Simple

Making a 'Creo Patcher'

When you're dealing with large userbases, it's often easier to wrap a standardized install up and deploy that rather than relying on the mismash of clicks that users tend to do. Over the years the Creo installation package has got freaking massive. I've come down these days on NOT running the installer on each users machine due to an unbeleivable amount of users not having the correct rights and so on. Instead I make an 'image' of Creo per OS platform (32bit Windows, 64bit Windows) and then have a custom AutoIT script on the users end suck that image onto the users client machine. I like to have the binaries on the users machine because that gives them the best possible performance on a day to day basis (installations are painfull and are done as infrequently as possible).

The problem remains however that once i've made a nice clean image for Creo, it usually takes some time to deploy the 4GB worth of files over the network to the client computer. We try to stage the image onto localized network drives but all of this is to be honest a total pain in the ass.

 

The IDEA

I decided that PTC's programmers no matter how incredible that have been over the years WOULD NOT likely change every single file in the install for each minor build. I also guessed that between minor builds they would NOT likely change the folder structure of the install beyond the BUILDCODE specific folders that we see.

So first I asked PTC Tech Support...

 

PTC Tech Support Answer

[Chris]: Hey I want to try something a bit different on this install thing cos the image size is so massive.

[Tech Support]: We cannot support any other method of installation other than as described in the...

[Chris]: Yeah I get it but it takes some folks hours, i'd like to improve their experience.

[Tech Support]: We cannot support any other method of installation other than as described in the...

[Chris]: Look I get it, what do you think unofficially.

[Tech Support]: We cannot support any other method of installation other than as described in the...

[Chris]: OK you're off my XMAS card list.

 

Similarity Analysis

Despite looking for freeware to compare two directories, I came up short. Instead I listed out what files existed in my M120 clean image (at DOS prompt dir /b /s >M120_Files.txt) and then wrote a small Script in AutoIT to run the compare per file.

The script wasn't very fast. This is principally because checking through files that were 100% the same was being done 1 byte at a time. I left it running overnight and it was done. I have no clear idea how many hours it took.

;read down the list of files
;assume we don't need to update
;make two paths
;check sizes different -> if yes needs updating
;	if no different ->check byte by byte
;		->if different flag file as needing to update
;			->dump into files to update.txt comma demlimit sizes of files also
$ListOfFilesToCheckAgainst = "Files_M120.txt";
$DateCode2 = "M090";

FileChangeDir ("d:\ptc");
Local $fileslist = FileOpen($ListOfFilesToCheckAgainst);
; Check if file opened for reading OK
If $fileslist = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
 EndIf

Local $filesDiff = FileOpen("Files_M120_Diff.txt", 10);
; Check if file opened for writing OK
If $filesDiff = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
 EndIf
 
 Local $filesSame = FileOpen("Files_M120_Same.txt", 10);
; Check if file opened for writing OK
If $filesSame = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf

;Read down the files list
; Read in lines of text until the EOF is reached
;$DoAFew = 5;
While 1
   Local $line = FileReadLine($fileslist)
   If @error = -1 Then ExitLoop
	;If $DoAFew==0 Then ExitLoop
   Local $sAttribute = FileGetAttrib($line);
   Local $result = StringInStr($sAttribute, "D")
   If $result == 0 Then
	  ;Compare File Sizes First
	  $file1 = $line;
	  $file2 = StringReplace($line, "M120", "M090", 0,0);
	  If FileExists ( $file2 ) Then
		 If CompareSize($file1, $file2)=="Same Size" Then
			
			If CompareBytes($file1, $file2)=="Same Bytes" Then
			   ;MsgBox(0, "Same:", $line )
			   FileWrite($filesSame, $line & "," & FileGetSize ( $file1 ) & @CRLF)
			Else
			   ;MsgBox(0, "Diff:", $file2 )
			   FileWrite($filesDiff, $line & "," & FileGetSize ( $file1 ) & @CRLF)
			EndIf
		 Else
			;MsgBox(0, "Diff:", $file2 )
			FileWrite($filesDiff, $line & "," & FileGetSize ( $file1 ) & @CRLF)
		 EndIf
	  Else
		 FileWrite($filesDiff, $line & "," & FileGetSize ( $file1 ) & @CRLF)
	  EndIf
	 ;$DoAFew = $DoAFew -1;
   EndIf
WEnd


FileClose($fileslist)
FileClose($filesDiff)

;;ENDOFPROGRAM


Func CompareSize($file1, $file2)
   $comparison = "Unknown";
   $s1 = FileGetSize ( $file1 );
   $s2 = FileGetSize ( $file2 );
   $d = $s1-$s2;
   If $d==0 Then
	  $comparison = "Same Size";
   Else
	  $comparison = "Diff Size";
   EndIf

   Return $comparison
EndFunc

Func CompareBytes($file1, $file2)
;CHECK FILES ARE DIFFERENT
Local $file_1 = FileOpen($file1, 16);
Local $file_2 = FileOpen($file2, 16);

; Check if file opened for reading OK
If $file_1 = -1 Then
   MsgBox(0, "Error", "Unable to open file.")
   Exit
EndIf
 
; Check if file opened for reading OK
If $file_2 = -1 Then
   MsgBox(0, "Error", "Unable to open file.")
   Exit
EndIf

; Read in 1 character at a time until the EOF is reached
$comparison = "Unknown";
While 1
   Local $chars_1 = FileRead($file_1, 1)
   Local $chars_2 = FileRead($file_2, 1)
   If @error = -1 Then ExitLoop
	   
   $result = StringCompare($chars_1, $chars_2, 1)
   If $result = 0 Then
	  $comparison = "Same Bytes";
   Else 
	  $comparison = "Diff Bytes";
	  ExitLoop
   EndIf
WEnd

FileClose($file_1)
FileClose($file_2)

Return $comparison;

EndFunc

 

Result of Analysis

The result was very encouraging, we could save 1.5GB worth of copying per client install.

The next step is to test a hacked together image to proove that there isn't some fatal logic flaw I've not considered. Since the script was checking at the byte level, I feel confident that Creo's binaries won't care about where i got them from.

 

Image Testing

After a couple of failed attempts to use Excel to build a batch file, I resorted to AutoIT again.

;Copy files to tempCreo2Parametric_m120* from M090
Local $files_to_reuse = FileOpen("D:\ptc\Files_M120_Same.txt", 16)

; Check if file opened for reading OK
If $files_to_reuse = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
 EndIf
 
; Read in one line at a time
While 1
   Local $line = FileReadLine($files_to_reuse)
   If @error = -1 Then ExitLoop

   ;Split line to get just the path out
   $file = StringSplit ( $line, "," )
   $sourcefile =  $file[1]
   $newfile = StringReplace($file[1], "creo2parametric", "tempCreo2Parametric")
   ConsoleWrite($sourcefile & " " & $newfile)
   FileCopy ( $sourcefile, $newfile, 9)
WEnd


FileClose($files_to_reuse)

;Copy files to tempCreo2Parametric_m120* from base image of M120
Local $files_to_reuse = FileOpen("D:\ptc\Files_M120_Diff.txt", 16)

; Check if file opened for reading OK
If $files_to_reuse = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
 EndIf
 
; Read in one line at a time
While 1
   Local $line = FileReadLine($files_to_reuse)
   If @error = -1 Then ExitLoop

   ;Split line to get just the path out
   $file = StringSplit ( $line, "," )
   $sourcefile =  $file[1]
   $newfile = StringReplace($file[1], "creo2parametric", "tempCreo2Parametric")
   ConsoleWrite($sourcefile & " " & $newfile)
   FileCopy ( $sourcefile, $newfile, 9)
WEnd


FileClose($files_to_reuse)

On initial inspection the images appear identical (as they should).

 

On initial inspection the image runs just like the kosher M120 install (as would the aforementioned logic result in).

 

Patch Builder

Goal is to make an install procedure that has a significant time benefit versus the current deployment method.

Strategy A: a) Copy M090 Files b) Copy from Network 'patch'.

Strategy B: a) Copy M090 Files b) Copy zipped Patch from network c) Extract patch into local.

My first hope was that i could feed Robocopy with a list of files. This it turns out isn't possible really and many internet leads ended up with me concluding that using the system copy commands was the preferred method. I did consider Robocopying M090 completely and then using a script to hack it but I generally prefer the concept of building things up. I also have since the inception of Windows 7 noticed that for some unknown reason deleting files seems to take a long time relative to my expectation. These beliefs drove me to instead do the following.

I wrote a small AutoIT routine to copy and paste files from M090 to a new M120 image. It took about 9 minutes which felt a bit dissappointing to be honest because a full RoboCopy from my local network drive takes about 9 minutes itself. Clearly the professional multi-threading of Robocopy is kicking my little scripts ass. I could have stopped here because technically if the user has M090 on their system then this procedure can be executed without downtime on their part. Nevertheless that seemed to be an excuse to me so instead I continued with my goal of making the 9 minutes something rather smaller.

At this point, I went off track a bit, I guess, due to the letdown. I decided to see what a Full NSIS installer would do. It took a long time for NSIS to build the almost 60k files into a single installer (I know the PTC folks probably thought about this a lot more than I have). Anyway although I was initially excited that the installer exe was ONLY a tiny 1.2GB (Compared to 4GB for my image), I didn't expect that expanding the compressed files would take so much CPU time and in the end it turned out that it took 8minutes and 40 seconds for me to run the installer from my disk. So another dead end was ran down.

I figured that the difference between my AutoIT script and Robocopy is that Robocopy is highly multi-threaded. Thus to match its performance I would need to multi-thread my file copying. Before I tried to write a 'chris threaded' M090 duplicate file copier I decided to check that the Difference could be dealt with in a reasonable manner. I started getting excited, the zipped 'patch' was only 760MB on my client disk. Testing indicated i could Robocopy this from the network drive in about 20 seconds. Because Windows 7 doesn't come with a command line unzipper tool, I found unzip.exe which unzipped it in 45 seconds (I think i would go back to NSIS to make a deployer though). This made me think that the chase was still alive. Assuming that I want to cut in half the current time. My target is to find a way to copy the M090 files over inside of 3 minutes (resulting in 4mins install versus 9mins). [Note: It turned out later on in testing that my patch was wrong and thus a least a part of my logic was flawed, luckily it didn't matter.]

 

This script builds multiple text files each with a portion of the same files list. It then launches a copier with each file into a seperate process.

;====================
;Copy_M090_Files_Threads
;====================
#include <Timers.au3>

Local $starttime = _Timer_Init()

;Copy files to tempCreo2Parametric_m120* from M090
Local $files_to_reuse = FileOpen("D:\ptc\Files_M120_Same.txt", 16)

; Check if file opened for reading OK
If $files_to_reuse = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf
 
$batch1 = FileOpen("D:\ptc\Files_M120_Same_b1.txt", 10) ; which is similar to 2 + 8 (erase + create dir)
If $batch1 = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf

$batch2 = FileOpen("D:\ptc\Files_M120_Same_b2.txt", 10) ; which is similar to 2 + 8 (erase + create dir)
If $batch2 = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
 EndIf
 
$batch3 = FileOpen("D:\ptc\Files_M120_Same_b3.txt", 10) ; which is similar to 2 + 8 (erase + create dir)
If $batch1 = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf

; Read in one line at a time
$Idx = 1
While 1
   Local $line = FileReadLine($files_to_reuse)
   If @error = -1 Then ExitLoop

   If $Idx<20001 Then
	  FileWriteLine ($batch1, $line)
   ElseIf $Idx<40001 Then
	  FileWriteLine ($batch2, $line)
   Else
	  FileWriteLine ($batch3, $line)
   EndIf
   
   $Idx = $Idx + 1
WEnd

FileClose($files_to_reuse)
FileClose($batch1)
FileClose($batch2)
FileClose($batch3)

ShellExecute("Copy_m090_files_Main.exe", "D:\ptc\Files_M120_Same_b1.txt m090 m120")
ShellExecute("Copy_m090_files_Main.exe", "D:\ptc\Files_M120_Same_b2.txt m090 m120")
ShellExecute("Copy_m090_files_Main.exe", "D:\ptc\Files_M120_Same_b3.txt m090 m120")

While ProcessExists("Copy_m090_files_Main.exe")
   Sleep(5000)
WEnd


ConsoleWrite(_Timer_Diff($starttime))
;--------------------------------

 

This script is the copier, it executes the copying from e.g. M090 to M120.

;====================
;Copy_M090_Files_Main
;====================
; $CmdLine[1] = e.g. "D:\ptc\Files_M120_Same.txt"
Local $files_to_reuse = FileOpen($CmdLine[1] , 16)

;Copy files to tempCreo2Parametric_m120* from M090
Local $BC_From = $CmdLine[2] 
Local $BC_To = $CmdLine[3] 
; Check if file opened for reading OK
If $files_to_reuse = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
 EndIf
 
; Read in one line at a time
While 1
   Local $line = FileReadLine($files_to_reuse)
   If @error = -1 Then ExitLoop

   ;Split line to get just the path out
   $file = StringSplit ( $line, "," )
   $sourcefile =  StringReplace($file[1], $BC_To, $BC_From) 
   $newfile = StringReplace($file[1], "creo2parametric", "tempCreo2Parametric")
   FileCopy ( $sourcefile, $newfile, 9)
WEnd

FileClose($files_to_reuse)
;--------------------------------

With 3 threads 4Mins 53Seconds was the result with 20% CPU consumed on my workstation.

With 6 threads 2Mins 17Seconds was the result with 50% CPU consumed on my workstation. Even though I could conceivably increase the number of threads again and consume more CPU's, this time is under my target so I'll continue forward with this routine for copying M090 re-useable content.

I established that copying the M120 Patch using Robocopy /FFT /S /MT /XX switches copied in 50 seconds which would beat the Zip copy plus unzip.[Note: It turned out later on in testing that my patch was wrong and thus this time was a bit flawed.]

I decided to join [fancy word for adding the below line] together the M090 Copy with the Patch Copy from Network (Strategy B) as a go-forward strategy.

ShellExecute("Robocopy", 
   "\\networkserver\M120PatchCreo2Parametric_m120 d:\ptc\tempCreo2Parametric_m120 /FFT /S /MT /XX")

The algorithm processed the 'install' in 2 Mins 57 Seconds. Catch is when i checked the Disk properties i have about 1500 files missing. Obviously somewhere along the way, some of the files are getting missed or the copies are failing. I think the next step is to put some error handling into the filecopy to see if we can try and avoid the blocks.

After some investigation it turns out i had made a mistake with my 'patch builder', i saw no failures in the M090 copy section. Rather somehow I had messed up checking that part and was missing thousands of those files. I recreated the patch (took only 90 seconds for the corrected Patch Builder script to copy all 3484 different files from the M120 image to a patch folder) and running the full installer this time with the correct patch got a time of 2 Mins 27 Seconds (since RoboCopy from network wasn't the issue, I can only imagine the difference is that for this test I had shut down Creo whereas in previous runs, I had Creo M090 actively running). The image appears to boot and is exactly the same number of files and filesize as my original M120 image (the goal).

 

Final Analysis

Current Method times: 9 Minutes 6 seconds to Robocopy from Local Drive.

Strategy A times: 2.5 to 3 Minutes with LAN to Network drive.

Strategy B times: [Estimated] 3 Minutes on LAN.

Other Benefits/Issues: Deployment of media could be done smoother, potentially less space required on network drives. True install failures could be harder for an admin to diagnose/fix however.

 

Conclusion

I beleive it truly is feasible and beneficial to build a 'patcher' for enterprise Creo deployment.

Strategy B would likely benefit users situated in remote parts of the network. Strategy A would be the fastest method for users with a LAN connection to the shared network drive where we deploy Creo from. At least $10k per year could be saved in pure wait time. More likely the number would be of the order $50k due to the lengthy deployments our WAN users have to face.