It sounds like this is a proprietary db designed by the team that wrote the application. Storing the db (as a file, so far as Linux is concerned) may work without any corruption. It's worth a try testing it. (but do it on a test machine, not on any units in productive use)
As mentioned there can be locking issues, but you'll never know until you try it. I have set up many proprietary database-network aware apps for clients that are able to use a database stored on a Linux server and providing access over samba.
I've also tested many that just wouldn't work this way. Like I say, give it a try.
A couple pointers: always use share level access on the server. Also, set the permissions of the db file to "Vegas cat house" setting. (chmod 777)
Another thing I have discovered, and actually made the difference between being able to use a Linux server of not, is that minuscule timing issues can make the project look like a total failure. I have noted in a few instances that we needed to have a high end switch with enough ports to have every single client (workstation) attached at a single point.
Just a couple weeks ago we added a workstation to such a network, (Linux serving a db) and for convenience I wired it to a cheap wifi router that was attached to the main switch, that happened to be in this same room.
The application they are using would lock up within 20 minutes, and I mean frozen solid. You'd restart everything, fire up the app, it's work for a bit then lock again.
I ran about 100 feet of cat 5 wire to connect this new workstation directly to the central (cisco 36 port) switch. Problem solved, it hasn't locked once since.
BTW I note this is NOT an "application server" model. The clients are all installed separately on each windows workstation, it's only the database they get from the server, via browsing for the file over the network.
Most, if not all of these types of applications 'remember' where they are getting the database from, the user does not have to "browse" the network every start up, just the first time.
I have set up two (IIRC) true application servers using Linux as the server, but these were rather odd-case apps. What I did was install the app on a windows machine first then copied the whole file structure over to Linux, then when the app goes to run on a windows client you start the process by executing the app on the windows machine, but the executable is first retrieved from the Linux machine.
As anyone from the old school may have guessed, these two apps were ancient 16 bit apps written with no concept of a "registry" whatsoever. The kind that don't really "install," but rather stuff their environment in a folder with an ".exe" to orchestrate the whole deal.
I have yet to get any critical windows network-based app running off a Linux server to any usable degree. (that's using wine) I have set up windows VMs on a Linux server to serve apps, but that's cheating. :) Actually, there are still "Linux benefits," like being able to back up the whole VM, data and all, nightly or whenever with rsync.
Back to setting up the test scenario... The issue throwing wrenches in the gears is most often very low tolerance on the cooperative timing these apps engage, something I have little doubt a windows server would automagically mitigate by itself. But if the goal is to use Linux, which has substantial upside, then having a top shelf switch is a must.
The upside, btw is it's vastly easier to back the dang thing up, and the whole backup system is more reliable than all but the most expensive commercial products for windows. By avoiding the windows lock (using NTFS) you can even mirror the file to a backup location in real time, if you want to get real anal about it.
Usually I set clients up with a USB drive attached to the server that gets a 'snapshot' of the db every hour or two using rsync. I also set most of them up with an automated backup nightly to remote location using rsync over ssh. (plus put ssh on a non-standard port, disable version 1 and set up password-less keys)
You can waste a ton of time testing this kind of thing before concluding it just ain't gonna work. But I find it's worth it, if you do discover a given scenario is going to work. For example licensing issues go away except so far as any individual workstation's version of windows, which is usually a "done deal" by simple virtue of buying the computer.
But windows client licensing limits the number of machines it can connect to to 10. If your "server" is just an XP machine you've relegated to the task, then when your customer grows to the point they add that 11th machine, they discover the hard way that they can't use it... the "server" will not accept another client connection.
You fix that by throwing a lot of money at microsoft, or by deploying Linux in the server role. All the risk is on me when I research a given situation, that is I might spend a lot of time only to conclude Linux cannot be an option. But when it is, the customers are happier than they'll ever know. They're not out a lot of money, the system is more stable and reliable, and they have options not available to them previously (or if they go with a windows server product) for free, eg a VPN server, web server (think "company wiki") and a number of other possibilities, which are now all profit to me because the only issue is my labor getting it set up.
I hope you're still following your thread here... =\ I'd encourage you to try to set this up and give it a through testing.