Suggestion how to approach this project.

PatrickStar

PatrickStar

New member
Jan 18, 2015
143
2
0
Spain
Hi all.

I am looking to do some programming. I have been programming since a while - but I did that in a very specific niche using very specific tools and I never did web apps (just websites). I know some html5 and Python.

So to hit my new year resolution I need to create web app this year.

Here is something I would like to do - Scrumwise - Features - I am looking to do simplified version of it. Usual drag and drop and collaboration.

The trick is - I want to do it in a flat file system. No databases at all.
It should be also easy to update it globally and as secure as possible.

So what should I use to create this?

Can I do this with html5 for front and Python for backend?
Any advice and tips when working with flat file system?

Thanks!
 


Look into git based wiki solutions. Here are some:

https://github.com/gollum/gollum (powers github wikis)

https://github.com/scragg0x/realms-wiki (immature but very impressive)

Using the filesystem for persistent storage makes sense when the backend content is fully editable by humans, and lends itself well to the change-tree pattern of git. Wikis and content sites are a good use for it. The downside is frequent editing is expensive and does not always propagate immediately.

If you want to use the filesystem for this project I recommend creating a "cards" directory. Within that, you can organize subdirectories to your own specification, but every card should go into this directory. To create a card, make a new file, edit it, commit it, and push it to "draft" branch to edit or "master" branch to publish. Note that a HTML UI + Javascript call to Python API could do this for you.

Write a build script to parse the files in the "cards" directory. You can set the spec for how cards should be arranged, and any variables they should include. You can do things like "Title=Title of card" and parse that. The build script should load all the file data into memory, then serve a flask app, then use wget or flask-freeze to generate a static build of the site.

Use git to register a post-commit hook that runs the build script and deploys the resulting static site to a CDN.

As you can see, this solution makes sense for something like a community wiki, where files are edited locally by users, relatively infrequently, and do not depend on constant consistency. However, a scrum system necessarily needs to be realtime. Changes are frequent and need to propagate immediately. Also, it's unclear what advantages you gain from storing the content in a filesystem instead of a database. I would therefore avoid this approach and use a database. If that is too much hassle, try something like firebase or meteor.

Flask is good.
 
If you're looking to build a web app in python, django is a good choice. It provides a lot of standard functionality for you, so you don't have to reinvent wheels that have been invented several times. It will also abstract some of the lower level things that webapps need to do, so it should be quicker to pick up and get something up and running with than raw python.

You don't mention why you don't want to use a database, but I'm going to guess that it's because haven't worked with databases and think it will save time. If that's the case, I think you will find that there are a lot of problems you will come across with a file-based solution that you will spend a lot of time solving that you would get "for free" from a db. For example: ensuring data consistency, concurrency, and efficient searching are things any decent db are very good at, but that you would have to build yourself with a file-based approach.
 
Hi guys, thanks for all the answers, much, much appreciated.

Why flat filesystem? Well it is a part of challenge. Every year I challenge myself to something ridiculous difficult and try to achieve it. Flat filesystem seems like something growing and next year I want to start a startup, that will require me to know all the cons and pros of this format - so this is a perfect training ground for me.

Also I have worked with databases before and I am aware they are the easy way to go, but its good to challenge yourself, right?

So I have started already, I am doing some html5 front end atm - it seems easier than I thought. I will go completely stupid approach - front->back->front - where I start with creating front end, then back end, then redoing front end. I think this way I will learn more and will be able to recognize my own mistakes and fix them.

I have spent already some hours on the project and I think it will require 500 hours in total to finish - we will see though as I can run in to bigger troubles than I expect to.

I am not dependent on this project success so I don't mind if this will run in to troubles - even better, at least I will learn more.
 
Put it this way: your frontend code should be independent of your backend. Whether you choose files or database does not impact your frontend code.

Your frontend code should call a backend api (start with something simple, like bottle in Python or Sinatra in ruby). The api should query the file system or database. If you choose file system, keep in mind that file system operations are far more expensive than database operations, so you should avoid querying or editing the file system with every request. If you must do this, then you need to generate periodic builds of your website and deploy it as a static tarball to S3 or a CDN. That way every request will simply download files from amazon instead of querying your file system.

If you're scratching your head wondering what I am saying, you should use a database.

But I think your idea of starting with a frontend is good. Do that. It will motivate you and give clear direction for what you need from the backend. For now I would just do simple HTML with jquery calls to a backend Python bottle script with a few endpoints.
 
chatmasta said:
Put it this way: your frontend code should be independent of your backend. Whether you choose files or database does not impact your frontend code.

Your frontend code should call a backend api (start with something simple, like bottle in Python or Sinatra in ruby). The api should query the file system or database. If you choose file system, keep in mind that file system operations are far more expensive than database operations, so you should avoid querying or editing the file system with every request. If you must do this, then you need to generate periodic builds of your website and deploy it as a static tarball to S3 or a CDN. That way every request will simply download files from amazon instead of querying your file system.

If you're scratching your head wondering what I am saying, you should use a database.

But I think your idea of starting with a frontend is good. Do that. It will motivate you and give clear direction for what you need from the backend. For now I would just do simple HTML with jquery calls to a backend Python bottle script with a few endpoints.
Click to expand...

Sounds like a plan, I will sit more on it next week when I have a bit more time then. Thanks for pointing me towards some direction!
 
I actually like the front end first approach a lot, especially for something like this where you don't have any pages that need to be indexed. I'd actually go for Angular and probably Bootstrap. Angular needs more javascript code, but the advantage is you end up with a very stable front end that's way easier to manage in the long run. Bootstrap just has nice components that are always better than anything you think of on your own.

The thing about challenging yourself with flat files is it's really not a challenge. You wind up having to code a lot of the functionality that your DB will take care of, but it's not code you'll learn anything from. It's just a lot of really simple code that will explode whenever you don't get it exactly right. I hate relational databases but I'd actually go with one for this. The schema will be simple but your entities are going to have a whole mess of things that are calculated.

If you wanna challenge yourself with something that's tedious but actually well worth it, why not go test driven for the whole project?
 
Yes, you can use HTML5 and CSS for your FRONTEND. For me, PHP is easier to use than any programming language for BACKEND. And use a database, a flat filesystem is not a good idea.
 
Suggestion how to approach this project

If you want to achieve success on your published project, try to publish it on Kickstarter. As per my understanding, Kickstarter is the best website to complete our mission of bringing the creative projects to life. It helps in backing up the published projects by increasing the backers count. Even I tried, by submitting an article on Kickstarter. And guess was, it has increased the count.
 
Abstract you model for easily swappable file-system/database storage backends.
 
You must log in or register to reply here.
Top Bottom
Back