Initial Import for DbUp

March 4, 2025    Development Database DevOps

Initial Import for DbUp

I’m looking into using DbUp for it’s migration approach to databases.

I’ve found that the initial import process isn’t documented so I did some research.

Organization

Dasith gives a helpful walkthrough and code sample.

I like his approach of having a Migrations folder and different folders for StoredProcs and Functions. This allows for code review of the changed StoredProcs and Functions which are replaced everytime. We can’t do this with Views as that would have a major re-indexing performance hit on the database. His code shows how to run each seperately.

Initial Import Process

The GitHub issue comment gave the next clue

First, run a test run with a restored development database first. Gain confidence before going to production.

Simple approach

  1. Script the Create database (Tasks > Generate Scripts) into a new sql file in Git, but not in your migrations folder. This will be for records only or if you need to create a new database.

  2. Split the StoredProcs and Functions into individual files to enable good code reviews

  3. Each of these should start with check for existence and drop if exists. IF EXISTS(SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[SP_GetMiles]') AND type IN(N'P', N'PC')) DROP PROCEDURE [dbo].[SP_GetMiles];

  4. This will be a lot of work, see the blog post I linked above for a hint

  5. You could choose to wait to make the StoredProc files until you modify one. That’s it, unless you want to test it out. Go ahead, you are working against your restored test copy.

  6. Optionally: Create a sample migration script for a test. This could be a new sproc. Note: this will always live in your source and SchemaVersions table.

  7. Run DbUp in you pipeline for the next change.

  8. Verify the changes from your pipeline

  9. Create a new migration script to delete the sproc.

the downside of this approach is that you will have to manually run the initial import script for any new versions of this database.

More complicated alternative approach

Note: I ended up going with the simple approach, so this is not fully vetted.

You will have to make sure the SchemaVersions tables is created and poplated before running DbUp. That makes this approach more risky, but will automatically create the database. You may have to add EnsureDatabase.For.SqlDatabase(connectionString);, but will that fight with the initial import script?

  1. Create the SchemaVersions table manually. I scripted out the table DbUp created for me in a fresh database.
CREATE TABLE [dbo].[SchemaVersions](
	[Id] [int] IDENTITY(1,1) NOT NULL,
	[ScriptName] [nvarchar](255) NOT NULL,
	[Applied] [datetime] NOT NULL,
 CONSTRAINT [PK_SchemaVersions_Id] PRIMARY KEY CLUSTERED 
(
	[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, OPTIMIZE_FOR_SEQUENTIAL_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]
GO
  1. Script the Create database (Tasks > Generate Scripts) into a new Migrations/00001_InitialImport.sql file
  2. Split the StoredProcs and Functions into individual files to enable good code reviews
  3. Each of these should start with check for existence and drop if exists. IF EXISTS(SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[SP_GetMiles]') AND type IN(N'P', N'PC')) DROP PROCEDURE [dbo].[SP_GetMiles];
  4. This will be a lot of work, see the blog post for a hint
  5. Manually insert 00001_InitialImport.sql into the SchemaVersions table
INSERT INTO [dbo].[SchemaVersions]
           ([ScriptName]
           ,[Applied])
     VALUES
           ('00001_InitialImport.sql'
           , '2025/02/25')
GO
  1. Run DbUp from Visual Studio with breakpoints
  2. the console output will show no scripts were applied
  3. Verify the database copy
  4. Take a copy of the real database
  5. repeat for the real database
  6. Verify the real database

My C# DbUp Code

// run the DbUp Migration scripts in the Scripts Folder
// change the connection string in the launchSettings.json for local development
//  in this case, I am using WithScriptsFromFileSystem so I could support multiple databases with this same exe. The WithScriptsFromFileSystem option is also available.

using DbUp.Helpers;
using DbUp;
using CommandLine;

var result = Parser.Default
            .ParseArguments<Options>(args)
            .MapResult(Run, Error);

return result;

static int Run(Options opts)
{
    try
    {
        Console.WriteLine($"Beginning Database Migration with these options {opts}");

        var connectionString = "";
        var databaseName = "";
        if (!string.IsNullOrWhiteSpace(opts.ConnectionString))
        {
            connectionString = opts.ConnectionString;
            if (!connectionString.Contains("Database="))
            {
                throw new Exception($"Connection String must contain `Database=`, it was {connectionString}");
            }

            var startIndex = connectionString.IndexOf("Database=") + 9;
            var colonIndex = connectionString[startIndex..].IndexOf(';');
            databaseName = connectionString.Substring(startIndex, colonIndex);
        }
        else
        {
            throw new Exception("--connectionString is required");
        }

        var scriptsPath = string.Empty;
        scriptsPath = !string.IsNullOrWhiteSpace(opts.ScriptsPath) 
            ? opts.ScriptsPath 
            : throw new Exception("--scriptsPath is required");

        var baseNamespace = typeof(Program).Namespace + databaseName;

        // Pre deployments - if we want to add that capability
        // WriteToConsole("Start executing predeployment scripts...");
        // string preDeploymentNamespace = baseNamespace + ".PreDeployment";
        // var preDeploymentScriptsPath = scriptsPath + "/Migrations";
        // RunMigrations(connectionString, preDeploymentNamespace, preDeploymentScriptsPath, true);

        // Migrations
        WriteToConsole("Start executing migration scripts...");
        var migrationScriptsPath = scriptsPath + "\\Migrations";
        RunMigrations(databaseName, connectionString, migrationScriptsPath, false);

        // Post deployments
        WriteToConsole("Start executing postdeployment scripts...");
        string postDeploymentScriptsPath = scriptsPath + "\\PostDeployment";
        RunMigrations(databaseName, connectionString, postDeploymentScriptsPath, true);
        return 0;
    }
    catch (Exception e)
    {
        WriteToConsole(e.Message, ConsoleColor.Red);
        return -1;
    }
}

static int RunMigrations(string databaseName, string connectionString, string scriptsPath, bool alwaysRun = false)
{
    WriteToConsole($"Executing scripts in {scriptsPath} for {databaseName}");

    var builder = DeployChanges.To
        .SqlDatabase(connectionString)
        .WithScriptsFromFileSystem(scriptsPath)
        .LogToConsole();

    builder = alwaysRun ?
         builder.JournalTo(new NullJournal()) :
         builder.JournalToSqlTable("dbo", "SchemaVersions");

    var executor = builder.Build();
    var result = executor.PerformUpgrade();

    if (!result.Successful)
    {
        throw new Exception(result.Error.ToString());
    }

    ShowSuccess();
    return 0;
}

static void ShowSuccess()
{
    WriteToConsole("Success!", ConsoleColor.Green);
}

static void WriteToConsole(string msg, ConsoleColor color = ConsoleColor.Green)
{
    Console.ForegroundColor = color;
    Console.WriteLine(msg);
    Console.ResetColor();
}

static int Error(IEnumerable<Error> errors)
{
    var result = -2;
    Console.WriteLine("errors {0}", errors.Count());
    if (errors.Any(x => x is HelpRequestedError || x is VersionRequestedError))
        result = -1;
    Console.WriteLine("Exit code {0}", result);
    return result;
}

class Options
{
    [Option('c', "connectionString", Required = true)]
    public string ConnectionString { get; set; } = string.Empty;

    // add an environment folder if/when needed based on https://dasith.me/2020/06/08/database-project-conversion-to-migrations/
    //[Option('e', "environment", Required = true)]
    //public string Environment { get; set; } = string.Empty;

    [Option('r', "reportPath", Required = true)]
    public string ReportPath { get; set; } = string.Empty;


    /// <summary>
    /// Run the migrations in the given path for the database
    /// </summary>
    [Option('s', "scriptsPath", Required = true)]
    public string ScriptsPath { get; set; } = string.Empty;

    public override string ToString()
    {
        return $"{ConnectionString} || {ReportPath} || {ScriptsPath}";
    }
}

I created a launchSettings.json file in Visual Studio for F5 running and use the command line args to run this in a pipeline. Using appsettings.json with transforms is another good option.

{
  "profiles": {
    "DbUpRunner": {
      "commandLineArgs": "--connectionString=\"Data Source=local\\DEVELOPER;Database=DbUp_Import;Integrated Security=True;Persist Security Info=False;Pooling=False;MultipleActiveResultSets=False;Connect Timeout=60;Encrypt=False;TrustServerCertificate=True\" --scriptsPath=\"C:\\git\\DbUp\\DbUpRunner\\DbUp_Import\"  --reportPath=\"C:\\git\\DbUp\"",
      "commandName": "Project"
    }
  }
}

A coworker is building an example inside of .Net Aspire , so check that out 🔭



Watch the Story for Good News
I gladly accept BTC Lightning Network tips at [email protected]

Please consider using Brave and adding me to your BAT payment ledger. Then you won't have to see ads! (when I get to $100 in Google Ads for a payout (I'm at $97.66!), I pledge to turn off ads)

Use Brave

Also check out my Resources Page for referrals that would help me.


Swan logo
Use Swan Bitcoin to onramp with low fees and automatic daily cost averaging and get $10 in BTC when you sign up.