XML Sitemap Generator side project - the journey so far

One of my long-time hobbies is programming, which I used to some degree commercially when I used to perform IT work for small businesses back in the early 2000s. It was post the dotcom boom, yet many businesses and organizations had yet to make the leap in to the online world (or even embrace digital).

Early days ASP and VB

During that period I did a lot of website work, involving both development and SEO. At that time, I possessed knowledge of VB 6.0 and some ASP from my university days, which was my starting point. In those days, JQuery wasn’t in existence, so while I delved into JavaScript programming, it was often a nightmare and something any sane person would avoid unless absolutely necessary.
Eventually, JQuery emerged, along with various plugins and frameworks. However, I primarily used JavaScript as a means to an end and NodeJS was still many years away.

Back then, the thriving open-source community we see today was still emerging, and the tooling wasn’t up to the standards we now enjoy. Consequently, I often found myself writing programs and scripts to accomplish tasks, feeds for content syndication, and data sharing, among other things.

So, in 2005 when XML Sitemaps came along, I found myself once again crafting scripts to automate their generation. Over time, this process matured until I decided to give it a UI, which is how G-Mapper, my XML Sitemap Generator for Windows (G-Mapper), was born. Check out that sexy UI!

I submitted this to online software websites and to my surprise it became a bit of a hit and I quickly had hundreds then thousands of users. As exciting as this was, it quickly became clear that while popular the software was buggy outside how I had used it and there was a demand for more features, including enhanced editing and image sitemaps. Over a period of time (probably months) I worked away on version 2.0, still in Visual Basic 6.0

Going online with Microsoft .Net

Version 2.0 proved to be just as popular, if not more so, garnering a reasonable amount of support from the community and also generating advertising revenue - a nice motivator to sustain the project. By this time, I had been immersed in VB.Net for quite a while and started to delve into C#.

Deciding that transitioning to an online service would likely be an easier option for maintenance, but knowing my codebase wasn’t ideal, I rewrote portions of the backend, porting them over to VB.Net, and developed a .Net Forms based front. The online XML sitemap generator was born, however, it quickly proved to be enormously problematic. Code written for a computer with ample resources does not scale well to accommodate many users. My service gained more popularity than I anticipated, creating a significant problem motivating me to optimize the backend “spider engine”.

The newly enhanced VB.Net spider engine served as the foundation for both the online version and Windows Version 3.0. This upgrade allowed me to streamline code and maintenance, although a complete transition to an online service had not been realized. I had started to experiment with moving the application online, but backend requirements and the need for more advanced front end frameworks were a step too given time I had available.

Version 3.0 of the Windows Software essentially marked a complete overhaul, departing from the legacy code by integrating the new VB.net spider engine with a brand new C# front end. The newer cleaner codebase made for easier maintenance, but there was still a lot to do.

As the online and Windows products gained popularity, I eventually began accepting donations to cover my costs and it did pretty well for a while. However as time went on advertising revenue declined, seemingly due to the introduction of GDPR. I decided to implement a small annual fee for using the service. The system proved successful and turned a modest profit, making it quite rewarding as a hobby project. Over the years though I wasn’t sure what to do with it, I maintained it, but it somewhat languished as I rolled in to a new day job.

Enter Javascript / Typescript

As an influence of my (non-programming) day job I had been dabbling with Javascript and nodeJs in my spare time to get to grips with the products and services. I already had some knowledge of JavaScript from years of tinkering, but coming from .Net I was quickly reminded of the horrors of the early 2000’s. It wasn’t until the early 2020’s that I embraced Typescript which immediately appealed to me due to its synergy with what I was familiar with from coding in .Net. Seeking a new project, and with only prototypes in mind, I toyed with the idea of rewriting my entire XML Sitemap Generator stack in JavaScript.

Excited by the diversity of Javascript, open source, the community and new frameworks every week, I made the decision to move away from Microsoft-specific technology and reinvent my online application. By this point the online app and marketing website had sprawled and become intertwined. I was keen to separate them, improve privacy and performance, and explore hosting options. I’ll write another blog about my journey with Hugo, Tailwind and Azure Static websites in the coming weeks.

Having quickly written most the back end in reusable Javascript modules with a cleaner model, but not fundamentally re-architecting, I still had to decide on a front end. After exploring several frameworks, they seemed complex for adding a touch of reactivity to pages, while leaving room for a richer application development in the future, then I stumbled upon SvelteKit. It appeared simple to get up and running, seamlessly integrating with my existing modules, without requiring a significant shift in my development mindset.

Finally after months of evenings and weekends coding, at the end of 2023 I published the new version.

I’ve thoroughly enjoyed working with SvelteKit; it’s been a breeze to get started and the community are really fantastic. That’s not to say there haven’t been some challenges and shortcomings along the way. While I have loved my experience with SvelteKit, there are certain limitations and issues I’ve encountered compared to other “frameworks”, notably express and routing. These will be the topic of a future blog post.

At this point due to time constraints I will stop supporting the Windows version, but maintain its presence for posterity. The website has had one final facelift and tidy up but my efforts will now focus on online services, with cross platform native client versions should there be a demand, but all based on a common codebase.

What’s the future?

Currently, I have some stabilizing and bug-fixing tasks on my plate. However, as any good developer knows, prioritizing rebuilding and embracing this weeks new framework is the top concern for customers …..

This has been a long-running project with likely more road ahead. I’ve already recognized that while the new JavaScript backend is performant compared to its predecessor, it could benefit from a re-architecture to enhance modularity, flexibility, and ease of maintenance. It currently involves long-running processes, a more distributed model is possible, albeit with some performance trade-offs. This revamped structure could also accommodate several other services I have in mind, some of which I experimented with on the old .Net backend but found challenging to implement and support due to poorer code architecture.

Furthermore, despite my enjoyment of working with SvelteKit, it does have some frustrating limitations and the recent introduction of Runes sent React vibes shivering down my spine. HTMX, the (old) new kid on the block, is something I’m eager to explore. It takes me back to the days of ASP, PHP, and MVC and would work really well with Hugo. I’ll likely delve more into this in future blog posts if I find the time in the coming weeks and months.

XML Sitemap Generator links

• Online XML Sitemap Generator
• Windows XML Sitemap Generator Download
• XML Sitemap Generators

Thoughts for another day

There are a whole host explorations in relation to this project which I plan to cover if / when I find the time to write about them.

• Migrating to Hugo, Tailwind and Azure Static Web Apps
• My excitement and frustration with SvelteKit
• The hosting and infrastructure side of the project
• Microsoft SQL Service migration to Mongo DB
• New service management regime built around Freshdesk
• SendGrid and Microsoft Exchange online.