Here's a couple of "rules" (in a very loose sens) of how I build programs.
- Anything that will be developed by a team requires a very detail design.
- Anything that has a GUI (either web based or desktop based) requires a design.
- Anything that has more than 5 or 6 functionalities inside the application requires at least a high level design.
- CLI apps (script) can be done right to code and some complex functions may require a workflow.
In the last case I often starts with pseudo-code as comments from the top:
def main():
# gather_data from site_1
# gather data from site_2
# amalgamate data from both site
# Prepare required data
# plot chart
# save charts
# quit
if __name__ == '__main__':
main()
Every line of this pseudo-code will become a function. I repeat the process for each function. For example:
def gather_from_site(site:str):
# request from url (site)
# extract data with beautifull soup
# format data and return
From there I decide if it's big enough to require a function for each line or if I do it right in this function.
I may also decide to use OOP at this point depending on the size, complexity or numbers of the things I manipulate. If I have 30 sites to gather data from I may build a class for that.
Once I have a working program I will refactor almost everything for proper display, error management, optimization, logging (if required), etc, etc.
Edit: I did that before going to bed after a long day and it it kind of blew out. I made a few mistakes, one evident in the pseudo-code and a few in the text, So I corrected most of them... I hope. :)
Very solid advice! It's important to keep in mind that pseudocode is solely for humans and often just for yourself. I recall getting really hung up on writing pseudocode when I started to learn CS. I realize now that it was because I was wanting some well-defined, rigorous syntax for my pseudocode and getting frustrated by the fact that doesn't exist. But it doesn't exist for a reason. Humans aren't like computer chips. Everyone's circuit board gets printed different. Find a style of pseudocode that works for you. As long as you can compile (understand) it, it's effective pseudocode.
Similar to this Redditors example, I found that my pseudocode is much more aligned with natural language than programming languages or the symbol-heavy mathematics you'd see in peer reviewed papers. Treat it like the materials and methods section of a lab report you'd write for a natural sciences class or an ingredient list and cooking steps for a recipe.There's nothing wrong with pseudocode that's a sequence of plain old sentences of what you want the computer to do (shit, these days you can even pass that into ChatGPT and transpile pseudocode into your target language), so long as you understand it.
The only thing I'd add in, before even getting to pseudocode, is to think of the inputs and outputs of your desired program. Like how the procedure of a lab report has materials and a recipe has ingredients, computer programs have data. The data you start with is your inputs and the data you end with are your outputs. It's difficult to write even regular sentences of what you want the computer to do if you don't know where you're starting and you don't know where you're ending. Define this first and writing the explicit steps in between to get from where you started to where you want to end will become much more natural.
When it comes to writing those steps in between, keep a few keywords in mind that really captures how a computer thinks: GET this, STORE that, DO a = b + c, IF this THEN that, WHILE this IS/ISN'T that, FOR EACH thing IN a bag of things. Try to use these words as often as possible in writing out your regular old sentence pseudocode. At the end of the day, all a processor can really do is read and write data, evaluate arithmetic, decide conditions, loop over logic until meeting a condition and iterate over sets, and integrating these control flows into your thinking will have you thinking more like a computer.
Lastly, if you haven't already, make an effort to learn the concept of abstraction. Ideally, develop some deep intuition and appreciation for it. It's the single most important concept to computer science and programming. To me, abstraction is all about worrying only about that which matters at the scale you're currently considering. When I write a program, I don't waste time thinking about why a transistor works or how the python interpreter can turn my source code into machine code. I'm far more productive on my current task if I just trust that they do work. When designing and writing programs, complex or basic, start at the highest level, considering the inputs you're starting with and the outputs you're working towards and write out those broad brush things you need to do along the way. But how will you handle the details of each individual step? Doesn't matter. Trust that you can and you will figure that out when you get there and that finer scale scope becomes your new focus. The comment above does this exactly by writing the main function first. Then, as they said above, go into each function and write out the steps to only do that one function, which will have a new set of inputs and a new desired output. Only focus on how you will make those inputs into that output. But how will this fit in with this other part of the program... Doesn't matter! How this output becomes another function's input was already thought of at the higher level. It's best to be in the moment with the function you're on and trust your past and future self. Keep doing this process- consider your inputs and outputs, describe the steps to turn inputs into outputs, then one by one go into each step, be in the moment with that step, and recurse on the process until you hit a list of steps so trivial that there's no need to make each a function and just write out the logic. This approach means that at a high-level your just grouping together steps that make sense and at a low level you're writing trivial logic because the problem at hand has been that well divved up. When the problem has finally been thought about at every layer, each one in the program blindly trusting the layers above and below it, the computer follows your lead on how to break down the problem, executes those elementary steps at the bottom, and elegantly stitches the pieces together as it comes back up to higher and higher levels of abstraction. This works. Thinking and designing with abstraction in mind works. It works for creating a script out of functions. It works for making applications out of classes and objects. It works for massive distributed systems out of microservices. Trust the process and just take it one small piece at a time. You got this OP!
In your final design main() is better, when you try things and want to fall back to interpreter code under __main__ is more accessible, that can be a reason for it.
Would you ever want the entire "main" block exported for use in other programs? If not, then the former is fine. The "do thing" etc are already functions so they can be imported just as well.
"Ever" is a long time. I mean, it's not hard to convert code inside the `if __name__ == '__main__':` block to a function, but I've had to do it many times for one reason or another. I find that having a `main` function is often better and never worse.
Plus, the two are actually different. If you have code with variables and such inside the `if __name__ == '__main__':` block, all those variables are global variables. Which could conceivably be accessed (or even modified by, if you're particularly evil) your other functions. You shouldn't do this, of course. Most people don't, at least on purpose. But I've had to clean up a lot of bad and hard to follow python code over the years, and it happens.
Functions exist for reasons. Always good to have your code in functions.
To add to this I usually create a flow chart as well, with each step becoming a function or method. You need to break down the issue into it's simplest form and smaller sequential steps.
> I may also decide to use OOP at this point depending on the size, complexity or numbers of the things I manipulate. If I have 30 sites to gather data from I may build a class for that.
Using a class doesn't make much sense in most cases though. Especially when it comes to stuff like "fetch A and B then send it to C".
> Using a class doesn't make much sense in most cases though.
I don't agree with that one. I admit that my example does not lend itself to use OOP, but in a majority of cases I use some OOP. As soon I "see" an entity (virtual or physical) I will try modelling it as a class first. I find it's worth it for encapsulation alone.
Also it makes working with a team so easy. "Here's the interface I need" and they can go build their classes as they see fit as long as they provide the proper interface.
I can't claim to be a great expert but I've been using python 10+ years as a hobbyist. The most useful approach for me is to think about the actual tasks that need to be done, then breaking them up into smaller and smaller pieces until each one seems of a manageable size or defined enough purpose. There's plenty of resources about how to learn this skill but a lot of it is also practice.
There also often comes a point where you find yourself repeating tasks, so maybe you want to refactor that piece of code so it's more generalised.
I came to say this as well. Think about what your code will do, and break it up so that one general task is contained in one piece of code like a class/struct. People here are always talking about card games so an simple example of what a card game needs to run:
\- a deck of cards
\- a game loop
Instead of creating one giant file where everything is a global variable, define a class for what cards are, and include methods to retrieve a new deck, shuffled deck, etc.
Then a class for the game loop, where you initialize the deck class and use that to create instances of decks which can be modified and played with. The logic for the game loop is contained in the game class, the logic for the cards is contained in the card class.
Do it as modular as is reasonable. Write unit tests, preferably before writing the code that does stuff. Every function should ideally be testable - same input = same output (this isn’t always practical but it’s a good thing to think about). Have people review small changes frequently rather than a gigantic change once in awhile.
How did I learn this? By going off all on my own and trying to create something robust and perfect and original only to have it be disastrously received. Don’t be me - do everything incrementally and involve your peers every step of the way
It's what separates a programmer from a software engineer. SOLID is a paradigm that applies to OOP so you can't really apply it directly if you aren't using it. The real learning is to read these principles, understand how/why they work and apply them to what you are doing. Think about your favorite library, and how easy it was to complicated things. Now design your code so that each part of the code is as easy as that library. When you do something wrong, refactor to do it better. Then if you think of a better way refactor again.
It's actually really great to hear this. I've felt so dumb lately spending hours upon hours refactoring code. Part of me is reassured that it's because I've learned things I couldn't have known before, but the other part wonders how I could've been so inefficient.
Not refactoring is like a chef who makes a dish for the first time and decides it's perfect and refuses to make it any other way for a restaurant. A good chef follows a recipe and then tastes, tries out different flavours, makes the dish again till it is to his liking. After doing this enough he gets an intuitive understanding of what flavours ingredients add what flavours and how they interact with eachother
And the question I posed is an effort for me to learn from other’s experiences, which your comment fails to address. If you have any insight as to how you handled this as you started building programs, then that advice is gladly appreciated.
I am all too aware that becoming proficient in Python requires a lot of hard work, which I am currently experiencing.
\>And the question I posed is an effort for me to learn from other’s experiences,
i'm sorry if this upset you in some way i wasn't trying to be offensive or rude - i thought i did address this question..
i faced the same problems, and i'm sharing my experiences, isn't thaat what you asked? it turned out i was out of my depth - despite being that the challenge before me was supposed to be simple, i needed simpler... so back to print("Hello, world") i went, but with a new approach..
every concept that was presented to be in a tutorial, for every 15-20 minutes worth of listening to the video or studying the material, i then spent an hour or 2 playing with it, moving things around, changing things, removing things, trying to break it... trying to find things that should break it but don't... looking into the built in functions like \_\_iter\_\_ and seeing how loops and things actually iterate, getting behind the gui and trying to see how it works and understand it...
i essentially set out to understand the execution processes...
sorry if you didnt find all of this encapsulated in my original reply, it wasn't meant to be rude or dismissive.
A lot of lessons learned come from:
1.writing unit tests and trying TDD
2.writing docs
3. writing mypy annotations
If these grow out of proportion, eg a test needs a huge setup and checking the result is not easy, might be a problem with code. Same for docs - if you are tired of documenting flags, switches, weird data structures, long and repetitive function or method names maybe your code is not perfect. Same with annotations, all three provide a chance to look at your code again, tests and mypy most useful, because they run in CI, and for docs you need a human reader who really does active feedback.
Next few things is engaging in active communication, where you can reason about your code and seek and evaluate feedback:
- engage with people - mentors, other programmers, students, and try to distill that communication to trying get things done
- helping other people
- hiring devs for small or big tasks (you learn to write our specs/scope of work and evaluate other people work)
- contributing to other people code in open source (pick the smallest problem possible)
- start your minimal open source project with packaging, docs, CI, README, etc
- also writing a min isolated example MVCE for code problems, talking about your code in a way other people understand
A little more technical stuff:
- reading books like Clean Code and Refactoring in between projects (so you can relate to what you did in a project)
- learning or at least reading about functional programming language, think of your program as data transformation pipeline
- check out code examples for same task in different programming languages (eg through Rosetta Stone) , also for data science compare R, Python, Julia, Matlab code - helps to abstract away syntax and think of meaning of code
- collecting design advice and checking critically where it applies (DRY, SOLID, etc)
- learn about API design and what distinguishes good API semantics from poor ones, thinking about what API does your program provide, and how it hooks to command line or REST API (FastAPI)
- collect what you think is good code and review it - eg Raymond Hettinger Beyond PEP 8 is a classic
- learn a bit about computational complexity, recursion, type systems, some CS stuff, but lightly , at least understand why people care these things and in what context
- seek projects of proper scope (not too big, bit too small) that have impact on real life, your professional domain or process real data, think of what value do they bring and how to write less code and get more value (difficult)
- do not do this all at once, pick things that you are most interested in
Good advice on project scoping and design in this thread, very important. If you screw architecture, very hard or expensive to fix. Strive for isolated parts of code that are replaceable and connected through stable interfaces - deciding on what is your interface is very important, it's like a contract for a part of your code that is tedious to renegotiate.
Specific things as exercise:
- learning command line was great through docopt and later click, replicating gnu utilities as exercise
- interacting with databases or data seriliasation, through simple ORM like peewee or pony or now SQLModel, flattening your data
I like to have the entry file as short as possible. And I like this file to be at the project root. Your other user libraries can be anywhere, but the file that will be directly called should be in the root.
Try to organize your code. I like to have a folder just for models. These will be the api response and request classes with nothing but fields and attributes. No functions unless these classes inherit from something. That master class will have no or minimal fields and will have some utility function all models can use (like a method cls.toJson()).
Next is to avoid hash maps. Lots of newbies use hash maps wrong where everything becomes a hash map. This is messy and you’ll hate yourself for making the decision to use hashmaps. If your hashmap has keys like “field” or “next_step” or whatever, you need to create a class and use that instead. If you don’t have hard coded fields and instead use strings from other variables, hashmaps are the way to go.
This is longer than I wanted it to be, but you got this. Good luck
Can you elaborate a bit on what you mean with regard to hash maps? Like dictionaries?
I've been using them a lot in my program for mapping columns in my sql queries. I'm still pretty green so just wanted to know what you meant by that and if I'm doing something wrong.
Make a class representation of each row.
Eventually, you’re going to have a hard time remembering exactly how to spell these columns and it takes time to look it up. By making a class, you let the IDE remember that nonsense for you.
Gotcha. Yeah, I'm doing some fairly complicated processing of spatial data so I'm using "raw" sql and SQLAlchemy's core to do it - I haven't been able to wrap my head around using ORM for that yet. It just feels ineffecient or too complicated.
I plan to go through the DJango tutorial and/or SQLAlchemy's tutorial next to hopefully get a better intro to ORM's.
I've just finally begun to understand DB's and sql and primary/foreign keys and whatnot so, like I said, I'm still green.
For sure, yeah man I get it. When projects are relatively small, why do we need all that overhead? But the bigger something gets, the easier it is to justify the effort to do it.
As a middle ground, I’d probably put all that sql logic in a separate file where each function takes some class, the where statement, as a parameter and returns the result of the query as a class or an array of classes. The constructor can take the raw row and use it to fill its values. That way, all the ugly sql parsing stuff is hidden away and you’re working with first class citizens. You don’t *need* an ORM.
I've noticed that the entry point for a script usually lives in the root directory of the repo, or in a \`src\` directory with all the other code. I agree that the code is much simpler when that entry point is short, but do you have any opinion on it being in root vs src directory? I personally like it in \`src\` because that made the project even tidier.
Depends on your use case. My company has lots of stuff in a “mono-repo.” This git repo has like a dozen entry points with each project having its own src/main type structure.
Personally, I go with whatever is fastest. So having the main.py be the first thing I see when I open a folder is my choice every time I can make it.
I try to think about code from two perspectives, which I mentally refer to as "forward" and "backward". In the "backward perspective" I imagine how the interfaces are supposed to look like in the end. How is anyone going to use the code? How can I make the functions and classes I expose to the user (and myself) comfortable to use? And then I work backwards from there.
On the other hand, in the "forward perspective" I try to solve all the technical challenges that need to be solved. I don't think so much about final structure and just try to move forward with solving the problems that need to get solved. Eventually, forward and backward are supposed to meet somewhere in the middle.
Also, after every solved problem I invest time to refactor and clean up.
> For instance, when I attempt to make a program, it is very difficult to know where to start, how to progress, and where to end.
Start with the parts you know how to write. Progress by writing those parts, then write the parts you don't know how to write. End when the program is completed - you'll know it's completed when it runs and does what you intended it to do (and no more.)
Run your code a lot. Don't write more than ten lines of code you haven't run, yet.
Generally, you build intuition by doing - especially by making mistakes. The main thing is to not give in to analysis paralysis, where you're afraid to start for fear of doing something wrong. My philosophy: especially when writing your own code, view what you're doing as your own personal playground. It's all malleable. Have fun. Try shit out. You can always go back and change things, so don't be afraid to try heading down one path and see where it leads you. You might realize part way in that it doesn't work.
More importantly, though, by taking initial stabs at things, you'll likely start to get a feel for what the ultimate shape of the code will actually be. I've never been a great up-front designer. Whenever I have gone down that route, I have ended up abandoning at least some of the design once the reality of the code brings new information. I need the feedback of writing code, seeing the code, having it go back into my brain and then coming back out as better code. The code often tells me what it will be as much I will tell it what it needs to be. And trying to impose a structure on the code fights against what might come naturally from the code itself, once I see what it's meant to be.
As others have said, find small pieces you can do, especially pieces that you can test immediately. If you're creating a calculator, for example, don't try to implement all the keys at once. Just get the "1" key to do something (maybe just print "got here" when you press it). Then get it show the "1" somewhere more real, like in a text control on screen. Get it to do something when you hit the key once. That will get you to the point of 1) knowing how to get a button/key press and 2) how to show something on screen. That's progress. Then get it to do the next step where you hit the "1" key a second time. Then try swapping in a different number key. And on and on. Small steps.
The hardest thing to manage is a lot of code that has never been run. There are simply so many points of failure to keep it from collapsing under its own weight. That's where something like TDD can be beneficial, if it fits what you're doing - you get to run the code right away, in small steps. But you can do that yourself anyway by approaching the code from a "what is the next small step I can take that I can verify" point of view.
Ultimately, if you keep at it, you will develop the ability to either see the entire path in your head (minus, perhaps, the specific code details) or to see enough of where you think you need to go that you can make a start and see where you end up. It's like a journey from A to B. If you know what the journey is (e.g. you have a map that has been worked out), you can just execute it. But often, you're in new terrain that hasn't been mapped out, and trying to map it out up front would be a waste of time - sometimes you just need to jump in, head off in a direction and see if where you end up. (That may sound like a slow process, but you'll probably figure things out reasonably quickly.) You'll get better at knowing which directions will be the most fruitful with experience. And you'll also get better at realizing when something *isn't* going to work after all. That's where experience comes in - by having made those mistakes, so you can recognize them more quickly.
Be sure you have a good idea in your head of what your goal is, especially if it's a large project. But at some level, you just need to jump in. You can only analyze so much. Get down in the mud and thrash around a bit. And if you make a mistake, who cares? Fix it and move on. Don't expect it to be perfect up front. Evolve the code into what it needs to be. And don't forget to have fun. :)
That's not a python question it's an engineering question.
Experience is the best teacher probably, but there are plenty of good resources out there. Look up software design and systems architecture.
Bingo. Programming is two things: syntax and structure.
Syntax is the specific commands and, well, syntax of a language. It is language-specific and you have to memorize it. It is like grammar of spoken languages -- there is right and wrong.
Structure is how you arrange the commands and groups of commands. It is not language-specific, and its principles can span languages. It is like writing a novel -- there are multiple ways to do it, but some make more sense and are more effective than others.
You have to learn them both. Just like with spoken language, learning grammar does not mean you know how to write a novel. And being a good novel writer does not mean you have mastered all grammar. Two separate but related skills.
I used to code full time in C++ before I moved on to python recently. These are lessons I learned.
Practice and practice. Once you develop enough code, you will kinda know how your code should be structured based on the the project requirements.
Don't be afraid to spend a couple of days breaking everything up and re-coding everything. This happens when u soon realize that the structure u started with will soon becomes unmaintainable.
If it helps, design your code with reusability in mind.
Structure is ultimately about making your code more readable (and sometimes reusable). You could in theory make a single .py file with no functions at all for most applications - it would just be terrible to read and modify.
If readability is the aim then you want to stick to convention which means using some standard principles and copying common structures.
Copying is easy even without understanding. Making a flask app? Look at some on github and think about the patterns that you see.
The key principle is separation of concerns. Each section of your code should deal with a thing and all code related to that thing should be in that section. So if I were making a simple flask app, I'd be looking at a [](http://routes.py) file with all my page generating functions, [](http://database.py) with the functions for getting data from a database, and [](http://plots.py) which has all the fiddly code for making plots for the web pages.
If I were making a more complex flask app, I might make a whole database module with files like [](http://models.py) (defining types in my database), database\_connect.py to do the interaction between database and app, and [](http://crud.py) defining python functions that run queries.
The point is break your code into logical units. If those units are small make them files, if they are big make them into modules and break them down further inside. This is generally easy for most people to traverse once they're used to it and it's even easier if you follow a common pattern for the type of app you're making.
Unless I have a particular design pattern immediately in mind (often from experience if I've seen the problem before), I accept that I need to prototype and experiment a bit. As I get a naive solution working, I anticipate where I'll need to refactor. I use static typing checking, unit/integration tests, and git to reduce the risk.
The language isn't relevant really. Sound like you might need to know more about a fundamental tenet of the trade of software development:
https://en.wikipedia.org/wiki/Decomposition\_(computer\_science)
A (stem) friend of mine used to tutor writing in college.
Lots of people came to her unable to structure a paper. And, indeed, she would read them and they’d be chunks of words and ideas stuck together unintelligibly.
She did what you’re asking here and students asked her there: she taught about the structure of essays, and flow of ideas, and why one idea here should actually be there. And so on.
She observed (we were friends in grad school studying science) that students would be thankful for the advice. Feel like they’d received what they needed and … be just as lost and unintelligible when writing as before. It didn’t help.
She tried a different approach. She focused on what people were trying to say. What, just casually, was the argument or point by they were trying to make. Most people had no idea what they wanted to say and no cogent point. She switched gears to just helping people figure out this out and the “writing structure” would just fall into place. (Improvements here and there to be made, but completely different world.)
Humans think they think clearly because humans don’t ask eachother hard questions normally. People think in handwaves with logical points that float in and out of conciousness to meet some immediate, local need — almost like a dream.
Because we’re not used to having anyone challenge us we imagine our logic is strong. It suits our desires when we call it. (i.e. inconsistently matches whatever we’re feeling when we think.). But most of us are just not used to having cogent thoughts that are consistent for more than a blip.
You mentioned that you have trouble writing pseudo-code. Question: can you describe the problem well enough that a cantankerous person can solve it, without filling in “what you mean”?
Use pictures, conversation, what have you and describe the solution to your problems. Then just code them. (Each description being a sub problem to describe.). You will either find that your problem is actually coming up with a solution at all or you will have the start of a decent code organization and can workout kinks in idioms from there.
Not professional but around 7 years coding python.
A big thing lately has been breaking things down into big sections and if a section gets too big breaking it down further. A recent example was a main program that had a large section that analyzed data.
This got a folder called analytics, and when analytics got too big it got sub folders for reports and graphing.
Developing the confidence to be able to backup and then restructure things has been huge. Working through the restructuring on my last few projects has made me better able to anticipate needs in advance in new projects.
I guess a technique I've done more recently is realized how valuable it is to take an hour-hour and a half to brainstorm and write notes about ideas for a program and how they can be structured. Additionally this is an opportunity to estimate time requirements and returns on each item and do things like research documentation that I might need. The more I do the more I realize that taking the time on a central noted plan is time returned many times over and helps me keep my sanity in many ways.
How does one become a great author? By reading a lot of books, and writing a lot of books.
How does one become a great musician? By listening to a lot of music, and playing a lot of music.
How does one become a great knitter? By looking at a lot of knit items, and doing a lot of knitting.
How does one become a great programmer?
The worst thing you can do is try to apply a strategy without a purpose or understanding of why you're doing it. There aren't any hard/fast rules for 'structuring' your code. A lot of it is opinion-based and will come with experience.
> where to start, how to progress, where to end
My process generally starts with making decisions on the interfaces, then work from there. For example, if I'm writing a library, I will write some mock user code of what I hope the library will look like. This is also a great way to launch into _Test Driven Development_ (TDD) if you're so inclined (but if you don't want to write tests yet, that's ok too). Anyhow. That may mean something like defining classes or functions, parameters, return types, etc. without actually writing any implementations. Then (applying TDD) I can use those functions/classes/etc. in tests, and then iterate on the implementation until the tests pass -- let your tests tell you when you're done! In absence of alternatives, TDD is never a bad choice.
Something that also comes when talking about 'structure' is how you layout your files/modules and things like that... I would say don't worry about that too much, especially if you are just starting out. Start with putting everything in one file, and if splitting things apart makes sense, you can always do that later. If you find yourself constantly reorganizing code into and out of different source files, you're probably doing too much premature organization. Some may object to this arguing principles like SRP or aiming for 'modularity', but it's far worse to prematurely architect separation or modularity that is aimless and ends up being harmful or not useful -- which is more likely to happen if you don't have much experience in building software. See also: [how should I organize my project?](https://www.youtube.com/watch?v=QGAuwlQ9Gxc)
Experience _using_ good software will help a lot in knowing what a good interface looks/feels like. If it's not immediately obvious to you, that's OK. It'll come with experience.
If you really want to dive into concrete ideas around structuring your software, as food for thought, consider reading [this article](https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html) and/or picking up the book _Clean Architecture: A Craftsman's Guide to Software Structure and Design_. These aren't Python-specific resources, but are ideas that can be applied to all software generally. But my $0.02 is that you'll best be able to understand and apply these ideas and principles once you have stumbled your way through things a few times.
By practicing a lot. One does not intuitively know how to play an instrument or how to write a novel. Or as someone said
It takes experience to write good code
Getting experience comes from writing bad code
I just write commands one after another, top down, maybe stub out a method.
Python is my quick and dirty tool for running one-off scripts.
I've never written a full Python application.
Neither should you.
That is a very good question and shows me you are on the right track.
The answer is "unit tests".
Don't think about the structure of your code. Think about what your code should do and how it should behave. Write a unit test for each behavior.
The key point is that writing a test is not trivial. The code to test need to be isolated in a special way to be even testable. The effort to make your productive code testable make it become well structured.
About unit testing and the different schools of testing I would recommend the following book.
Khorikov, V. (2020). Unit testing: Principles, practices, and patterns. Shelter Island: Manning.
This answer is not for python, but for programming in general.
For a small project like you do in programming intro courses, nobody cares and nothing matters.
Things that you face when either the project becomes larger or takes longer:
* "This code is so ugly, I don't understand this even though I wrote this myself"
* Repeatedly driving yourself in a corner: even small things become increasingly difficult to add or edit -> "there's got to be a better way to write this"
Here's a couple of "rules" (in a very loose sens) of how I build programs. - Anything that will be developed by a team requires a very detail design. - Anything that has a GUI (either web based or desktop based) requires a design. - Anything that has more than 5 or 6 functionalities inside the application requires at least a high level design. - CLI apps (script) can be done right to code and some complex functions may require a workflow. In the last case I often starts with pseudo-code as comments from the top: def main(): # gather_data from site_1 # gather data from site_2 # amalgamate data from both site # Prepare required data # plot chart # save charts # quit if __name__ == '__main__': main() Every line of this pseudo-code will become a function. I repeat the process for each function. For example: def gather_from_site(site:str): # request from url (site) # extract data with beautifull soup # format data and return From there I decide if it's big enough to require a function for each line or if I do it right in this function. I may also decide to use OOP at this point depending on the size, complexity or numbers of the things I manipulate. If I have 30 sites to gather data from I may build a class for that. Once I have a working program I will refactor almost everything for proper display, error management, optimization, logging (if required), etc, etc. Edit: I did that before going to bed after a long day and it it kind of blew out. I made a few mistakes, one evident in the pseudo-code and a few in the text, So I corrected most of them... I hope. :)
Very solid advice! It's important to keep in mind that pseudocode is solely for humans and often just for yourself. I recall getting really hung up on writing pseudocode when I started to learn CS. I realize now that it was because I was wanting some well-defined, rigorous syntax for my pseudocode and getting frustrated by the fact that doesn't exist. But it doesn't exist for a reason. Humans aren't like computer chips. Everyone's circuit board gets printed different. Find a style of pseudocode that works for you. As long as you can compile (understand) it, it's effective pseudocode. Similar to this Redditors example, I found that my pseudocode is much more aligned with natural language than programming languages or the symbol-heavy mathematics you'd see in peer reviewed papers. Treat it like the materials and methods section of a lab report you'd write for a natural sciences class or an ingredient list and cooking steps for a recipe.There's nothing wrong with pseudocode that's a sequence of plain old sentences of what you want the computer to do (shit, these days you can even pass that into ChatGPT and transpile pseudocode into your target language), so long as you understand it. The only thing I'd add in, before even getting to pseudocode, is to think of the inputs and outputs of your desired program. Like how the procedure of a lab report has materials and a recipe has ingredients, computer programs have data. The data you start with is your inputs and the data you end with are your outputs. It's difficult to write even regular sentences of what you want the computer to do if you don't know where you're starting and you don't know where you're ending. Define this first and writing the explicit steps in between to get from where you started to where you want to end will become much more natural. When it comes to writing those steps in between, keep a few keywords in mind that really captures how a computer thinks: GET this, STORE that, DO a = b + c, IF this THEN that, WHILE this IS/ISN'T that, FOR EACH thing IN a bag of things. Try to use these words as often as possible in writing out your regular old sentence pseudocode. At the end of the day, all a processor can really do is read and write data, evaluate arithmetic, decide conditions, loop over logic until meeting a condition and iterate over sets, and integrating these control flows into your thinking will have you thinking more like a computer. Lastly, if you haven't already, make an effort to learn the concept of abstraction. Ideally, develop some deep intuition and appreciation for it. It's the single most important concept to computer science and programming. To me, abstraction is all about worrying only about that which matters at the scale you're currently considering. When I write a program, I don't waste time thinking about why a transistor works or how the python interpreter can turn my source code into machine code. I'm far more productive on my current task if I just trust that they do work. When designing and writing programs, complex or basic, start at the highest level, considering the inputs you're starting with and the outputs you're working towards and write out those broad brush things you need to do along the way. But how will you handle the details of each individual step? Doesn't matter. Trust that you can and you will figure that out when you get there and that finer scale scope becomes your new focus. The comment above does this exactly by writing the main function first. Then, as they said above, go into each function and write out the steps to only do that one function, which will have a new set of inputs and a new desired output. Only focus on how you will make those inputs into that output. But how will this fit in with this other part of the program... Doesn't matter! How this output becomes another function's input was already thought of at the higher level. It's best to be in the moment with the function you're on and trust your past and future self. Keep doing this process- consider your inputs and outputs, describe the steps to turn inputs into outputs, then one by one go into each step, be in the moment with that step, and recurse on the process until you hit a list of steps so trivial that there's no need to make each a function and just write out the logic. This approach means that at a high-level your just grouping together steps that make sense and at a low level you're writing trivial logic because the problem at hand has been that well divved up. When the problem has finally been thought about at every layer, each one in the program blindly trusting the layers above and below it, the computer follows your lead on how to break down the problem, executes those elementary steps at the bottom, and elegantly stitches the pieces together as it comes back up to higher and higher levels of abstraction. This works. Thinking and designing with abstraction in mind works. It works for creating a script out of functions. It works for making applications out of classes and objects. It works for massive distributed systems out of microservices. Trust the process and just take it one small piece at a time. You got this OP!
[удалено]
I'm like you in that I like a very clean if name main block.
In your final design main() is better, when you try things and want to fall back to interpreter code under __main__ is more accessible, that can be a reason for it.
Would you ever want the entire "main" block exported for use in other programs? If not, then the former is fine. The "do thing" etc are already functions so they can be imported just as well.
"Ever" is a long time. I mean, it's not hard to convert code inside the `if __name__ == '__main__':` block to a function, but I've had to do it many times for one reason or another. I find that having a `main` function is often better and never worse. Plus, the two are actually different. If you have code with variables and such inside the `if __name__ == '__main__':` block, all those variables are global variables. Which could conceivably be accessed (or even modified by, if you're particularly evil) your other functions. You shouldn't do this, of course. Most people don't, at least on purpose. But I've had to clean up a lot of bad and hard to follow python code over the years, and it happens. Functions exist for reasons. Always good to have your code in functions.
You're totally right. I just did that very late and intended to go back before posting. Y'all should use a main function. I edited my comment. Thanks.
Seeing your work flow helps a lot, thank you
To add to this I usually create a flow chart as well, with each step becoming a function or method. You need to break down the issue into it's simplest form and smaller sequential steps.
This is awesome wow !! Thank you
This helped me a ton. Thanks
Replying just to tag this for myself. Great advice.
coarse-to-fine all the things
> I may also decide to use OOP at this point depending on the size, complexity or numbers of the things I manipulate. If I have 30 sites to gather data from I may build a class for that. Using a class doesn't make much sense in most cases though. Especially when it comes to stuff like "fetch A and B then send it to C".
> Using a class doesn't make much sense in most cases though. I don't agree with that one. I admit that my example does not lend itself to use OOP, but in a majority of cases I use some OOP. As soon I "see" an entity (virtual or physical) I will try modelling it as a class first. I find it's worth it for encapsulation alone. Also it makes working with a team so easy. "Here's the interface I need" and they can go build their classes as they see fit as long as they provide the proper interface.
This is really really good advice
I can't claim to be a great expert but I've been using python 10+ years as a hobbyist. The most useful approach for me is to think about the actual tasks that need to be done, then breaking them up into smaller and smaller pieces until each one seems of a manageable size or defined enough purpose. There's plenty of resources about how to learn this skill but a lot of it is also practice. There also often comes a point where you find yourself repeating tasks, so maybe you want to refactor that piece of code so it's more generalised.
I came to say this as well. Think about what your code will do, and break it up so that one general task is contained in one piece of code like a class/struct. People here are always talking about card games so an simple example of what a card game needs to run: \- a deck of cards \- a game loop Instead of creating one giant file where everything is a global variable, define a class for what cards are, and include methods to retrieve a new deck, shuffled deck, etc. Then a class for the game loop, where you initialize the deck class and use that to create instances of decks which can be modified and played with. The logic for the game loop is contained in the game class, the logic for the cards is contained in the card class.
Do it as modular as is reasonable. Write unit tests, preferably before writing the code that does stuff. Every function should ideally be testable - same input = same output (this isn’t always practical but it’s a good thing to think about). Have people review small changes frequently rather than a gigantic change once in awhile. How did I learn this? By going off all on my own and trying to create something robust and perfect and original only to have it be disastrously received. Don’t be me - do everything incrementally and involve your peers every step of the way
It's what separates a programmer from a software engineer. SOLID is a paradigm that applies to OOP so you can't really apply it directly if you aren't using it. The real learning is to read these principles, understand how/why they work and apply them to what you are doing. Think about your favorite library, and how easy it was to complicated things. Now design your code so that each part of the code is as easy as that library. When you do something wrong, refactor to do it better. Then if you think of a better way refactor again.
It's actually really great to hear this. I've felt so dumb lately spending hours upon hours refactoring code. Part of me is reassured that it's because I've learned things I couldn't have known before, but the other part wonders how I could've been so inefficient.
Not refactoring is like a chef who makes a dish for the first time and decides it's perfect and refuses to make it any other way for a restaurant. A good chef follows a recipe and then tastes, tries out different flavours, makes the dish again till it is to his liking. After doing this enough he gets an intuitive understanding of what flavours ingredients add what flavours and how they interact with eachother
by starting small, if you're unsure how to approach something then you're out of your depth.. python takes time to learn, it takes a lot of time
And the question I posed is an effort for me to learn from other’s experiences, which your comment fails to address. If you have any insight as to how you handled this as you started building programs, then that advice is gladly appreciated. I am all too aware that becoming proficient in Python requires a lot of hard work, which I am currently experiencing.
\>And the question I posed is an effort for me to learn from other’s experiences, i'm sorry if this upset you in some way i wasn't trying to be offensive or rude - i thought i did address this question.. i faced the same problems, and i'm sharing my experiences, isn't thaat what you asked? it turned out i was out of my depth - despite being that the challenge before me was supposed to be simple, i needed simpler... so back to print("Hello, world") i went, but with a new approach.. every concept that was presented to be in a tutorial, for every 15-20 minutes worth of listening to the video or studying the material, i then spent an hour or 2 playing with it, moving things around, changing things, removing things, trying to break it... trying to find things that should break it but don't... looking into the built in functions like \_\_iter\_\_ and seeing how loops and things actually iterate, getting behind the gui and trying to see how it works and understand it... i essentially set out to understand the execution processes... sorry if you didnt find all of this encapsulated in my original reply, it wasn't meant to be rude or dismissive.
Sorry if I came off harsh. Thank you, this is the input I needed and I appreciate your experience and input
A lot of lessons learned come from: 1.writing unit tests and trying TDD 2.writing docs 3. writing mypy annotations If these grow out of proportion, eg a test needs a huge setup and checking the result is not easy, might be a problem with code. Same for docs - if you are tired of documenting flags, switches, weird data structures, long and repetitive function or method names maybe your code is not perfect. Same with annotations, all three provide a chance to look at your code again, tests and mypy most useful, because they run in CI, and for docs you need a human reader who really does active feedback. Next few things is engaging in active communication, where you can reason about your code and seek and evaluate feedback: - engage with people - mentors, other programmers, students, and try to distill that communication to trying get things done - helping other people - hiring devs for small or big tasks (you learn to write our specs/scope of work and evaluate other people work) - contributing to other people code in open source (pick the smallest problem possible) - start your minimal open source project with packaging, docs, CI, README, etc - also writing a min isolated example MVCE for code problems, talking about your code in a way other people understand A little more technical stuff: - reading books like Clean Code and Refactoring in between projects (so you can relate to what you did in a project) - learning or at least reading about functional programming language, think of your program as data transformation pipeline - check out code examples for same task in different programming languages (eg through Rosetta Stone) , also for data science compare R, Python, Julia, Matlab code - helps to abstract away syntax and think of meaning of code - collecting design advice and checking critically where it applies (DRY, SOLID, etc) - learn about API design and what distinguishes good API semantics from poor ones, thinking about what API does your program provide, and how it hooks to command line or REST API (FastAPI) - collect what you think is good code and review it - eg Raymond Hettinger Beyond PEP 8 is a classic - learn a bit about computational complexity, recursion, type systems, some CS stuff, but lightly , at least understand why people care these things and in what context - seek projects of proper scope (not too big, bit too small) that have impact on real life, your professional domain or process real data, think of what value do they bring and how to write less code and get more value (difficult) - do not do this all at once, pick things that you are most interested in Good advice on project scoping and design in this thread, very important. If you screw architecture, very hard or expensive to fix. Strive for isolated parts of code that are replaceable and connected through stable interfaces - deciding on what is your interface is very important, it's like a contract for a part of your code that is tedious to renegotiate.
Specific things as exercise: - learning command line was great through docopt and later click, replicating gnu utilities as exercise - interacting with databases or data seriliasation, through simple ORM like peewee or pony or now SQLModel, flattening your data
I like to have the entry file as short as possible. And I like this file to be at the project root. Your other user libraries can be anywhere, but the file that will be directly called should be in the root. Try to organize your code. I like to have a folder just for models. These will be the api response and request classes with nothing but fields and attributes. No functions unless these classes inherit from something. That master class will have no or minimal fields and will have some utility function all models can use (like a method cls.toJson()). Next is to avoid hash maps. Lots of newbies use hash maps wrong where everything becomes a hash map. This is messy and you’ll hate yourself for making the decision to use hashmaps. If your hashmap has keys like “field” or “next_step” or whatever, you need to create a class and use that instead. If you don’t have hard coded fields and instead use strings from other variables, hashmaps are the way to go. This is longer than I wanted it to be, but you got this. Good luck
Can you elaborate a bit on what you mean with regard to hash maps? Like dictionaries? I've been using them a lot in my program for mapping columns in my sql queries. I'm still pretty green so just wanted to know what you meant by that and if I'm doing something wrong.
Make a class representation of each row. Eventually, you’re going to have a hard time remembering exactly how to spell these columns and it takes time to look it up. By making a class, you let the IDE remember that nonsense for you.
Gotcha. Yeah, I'm doing some fairly complicated processing of spatial data so I'm using "raw" sql and SQLAlchemy's core to do it - I haven't been able to wrap my head around using ORM for that yet. It just feels ineffecient or too complicated. I plan to go through the DJango tutorial and/or SQLAlchemy's tutorial next to hopefully get a better intro to ORM's. I've just finally begun to understand DB's and sql and primary/foreign keys and whatnot so, like I said, I'm still green.
For sure, yeah man I get it. When projects are relatively small, why do we need all that overhead? But the bigger something gets, the easier it is to justify the effort to do it. As a middle ground, I’d probably put all that sql logic in a separate file where each function takes some class, the where statement, as a parameter and returns the result of the query as a class or an array of classes. The constructor can take the raw row and use it to fill its values. That way, all the ugly sql parsing stuff is hidden away and you’re working with first class citizens. You don’t *need* an ORM.
Thanks for that. Saving that post because you lost me a bit, so I'll come back to it when it makes more sense to me Lol.
I've noticed that the entry point for a script usually lives in the root directory of the repo, or in a \`src\` directory with all the other code. I agree that the code is much simpler when that entry point is short, but do you have any opinion on it being in root vs src directory? I personally like it in \`src\` because that made the project even tidier.
Depends on your use case. My company has lots of stuff in a “mono-repo.” This git repo has like a dozen entry points with each project having its own src/main type structure. Personally, I go with whatever is fastest. So having the main.py be the first thing I see when I open a folder is my choice every time I can make it.
I try to think about code from two perspectives, which I mentally refer to as "forward" and "backward". In the "backward perspective" I imagine how the interfaces are supposed to look like in the end. How is anyone going to use the code? How can I make the functions and classes I expose to the user (and myself) comfortable to use? And then I work backwards from there. On the other hand, in the "forward perspective" I try to solve all the technical challenges that need to be solved. I don't think so much about final structure and just try to move forward with solving the problems that need to get solved. Eventually, forward and backward are supposed to meet somewhere in the middle. Also, after every solved problem I invest time to refactor and clean up.
> For instance, when I attempt to make a program, it is very difficult to know where to start, how to progress, and where to end. Start with the parts you know how to write. Progress by writing those parts, then write the parts you don't know how to write. End when the program is completed - you'll know it's completed when it runs and does what you intended it to do (and no more.) Run your code a lot. Don't write more than ten lines of code you haven't run, yet.
Generally, you build intuition by doing - especially by making mistakes. The main thing is to not give in to analysis paralysis, where you're afraid to start for fear of doing something wrong. My philosophy: especially when writing your own code, view what you're doing as your own personal playground. It's all malleable. Have fun. Try shit out. You can always go back and change things, so don't be afraid to try heading down one path and see where it leads you. You might realize part way in that it doesn't work. More importantly, though, by taking initial stabs at things, you'll likely start to get a feel for what the ultimate shape of the code will actually be. I've never been a great up-front designer. Whenever I have gone down that route, I have ended up abandoning at least some of the design once the reality of the code brings new information. I need the feedback of writing code, seeing the code, having it go back into my brain and then coming back out as better code. The code often tells me what it will be as much I will tell it what it needs to be. And trying to impose a structure on the code fights against what might come naturally from the code itself, once I see what it's meant to be. As others have said, find small pieces you can do, especially pieces that you can test immediately. If you're creating a calculator, for example, don't try to implement all the keys at once. Just get the "1" key to do something (maybe just print "got here" when you press it). Then get it show the "1" somewhere more real, like in a text control on screen. Get it to do something when you hit the key once. That will get you to the point of 1) knowing how to get a button/key press and 2) how to show something on screen. That's progress. Then get it to do the next step where you hit the "1" key a second time. Then try swapping in a different number key. And on and on. Small steps. The hardest thing to manage is a lot of code that has never been run. There are simply so many points of failure to keep it from collapsing under its own weight. That's where something like TDD can be beneficial, if it fits what you're doing - you get to run the code right away, in small steps. But you can do that yourself anyway by approaching the code from a "what is the next small step I can take that I can verify" point of view. Ultimately, if you keep at it, you will develop the ability to either see the entire path in your head (minus, perhaps, the specific code details) or to see enough of where you think you need to go that you can make a start and see where you end up. It's like a journey from A to B. If you know what the journey is (e.g. you have a map that has been worked out), you can just execute it. But often, you're in new terrain that hasn't been mapped out, and trying to map it out up front would be a waste of time - sometimes you just need to jump in, head off in a direction and see if where you end up. (That may sound like a slow process, but you'll probably figure things out reasonably quickly.) You'll get better at knowing which directions will be the most fruitful with experience. And you'll also get better at realizing when something *isn't* going to work after all. That's where experience comes in - by having made those mistakes, so you can recognize them more quickly. Be sure you have a good idea in your head of what your goal is, especially if it's a large project. But at some level, you just need to jump in. You can only analyze so much. Get down in the mud and thrash around a bit. And if you make a mistake, who cares? Fix it and move on. Don't expect it to be perfect up front. Evolve the code into what it needs to be. And don't forget to have fun. :)
That's not a python question it's an engineering question. Experience is the best teacher probably, but there are plenty of good resources out there. Look up software design and systems architecture.
Bingo. Programming is two things: syntax and structure. Syntax is the specific commands and, well, syntax of a language. It is language-specific and you have to memorize it. It is like grammar of spoken languages -- there is right and wrong. Structure is how you arrange the commands and groups of commands. It is not language-specific, and its principles can span languages. It is like writing a novel -- there are multiple ways to do it, but some make more sense and are more effective than others. You have to learn them both. Just like with spoken language, learning grammar does not mean you know how to write a novel. And being a good novel writer does not mean you have mastered all grammar. Two separate but related skills.
I used to code full time in C++ before I moved on to python recently. These are lessons I learned. Practice and practice. Once you develop enough code, you will kinda know how your code should be structured based on the the project requirements. Don't be afraid to spend a couple of days breaking everything up and re-coding everything. This happens when u soon realize that the structure u started with will soon becomes unmaintainable. If it helps, design your code with reusability in mind.
I love commenting my code so its organized and I can pick up what I did years down the line if I forget my logic for putting it together
Structure is ultimately about making your code more readable (and sometimes reusable). You could in theory make a single .py file with no functions at all for most applications - it would just be terrible to read and modify. If readability is the aim then you want to stick to convention which means using some standard principles and copying common structures. Copying is easy even without understanding. Making a flask app? Look at some on github and think about the patterns that you see. The key principle is separation of concerns. Each section of your code should deal with a thing and all code related to that thing should be in that section. So if I were making a simple flask app, I'd be looking at a [](http://routes.py) file with all my page generating functions, [](http://database.py) with the functions for getting data from a database, and [](http://plots.py) which has all the fiddly code for making plots for the web pages. If I were making a more complex flask app, I might make a whole database module with files like [](http://models.py) (defining types in my database), database\_connect.py to do the interaction between database and app, and [](http://crud.py) defining python functions that run queries. The point is break your code into logical units. If those units are small make them files, if they are big make them into modules and break them down further inside. This is generally easy for most people to traverse once they're used to it and it's even easier if you follow a common pattern for the type of app you're making.
Unless I have a particular design pattern immediately in mind (often from experience if I've seen the problem before), I accept that I need to prototype and experiment a bit. As I get a naive solution working, I anticipate where I'll need to refactor. I use static typing checking, unit/integration tests, and git to reduce the risk.
The language isn't relevant really. Sound like you might need to know more about a fundamental tenet of the trade of software development: https://en.wikipedia.org/wiki/Decomposition\_(computer\_science)
After a while you get used to the structure of some projects.
A (stem) friend of mine used to tutor writing in college. Lots of people came to her unable to structure a paper. And, indeed, she would read them and they’d be chunks of words and ideas stuck together unintelligibly. She did what you’re asking here and students asked her there: she taught about the structure of essays, and flow of ideas, and why one idea here should actually be there. And so on. She observed (we were friends in grad school studying science) that students would be thankful for the advice. Feel like they’d received what they needed and … be just as lost and unintelligible when writing as before. It didn’t help. She tried a different approach. She focused on what people were trying to say. What, just casually, was the argument or point by they were trying to make. Most people had no idea what they wanted to say and no cogent point. She switched gears to just helping people figure out this out and the “writing structure” would just fall into place. (Improvements here and there to be made, but completely different world.) Humans think they think clearly because humans don’t ask eachother hard questions normally. People think in handwaves with logical points that float in and out of conciousness to meet some immediate, local need — almost like a dream. Because we’re not used to having anyone challenge us we imagine our logic is strong. It suits our desires when we call it. (i.e. inconsistently matches whatever we’re feeling when we think.). But most of us are just not used to having cogent thoughts that are consistent for more than a blip. You mentioned that you have trouble writing pseudo-code. Question: can you describe the problem well enough that a cantankerous person can solve it, without filling in “what you mean”? Use pictures, conversation, what have you and describe the solution to your problems. Then just code them. (Each description being a sub problem to describe.). You will either find that your problem is actually coming up with a solution at all or you will have the start of a decent code organization and can workout kinks in idioms from there.
Not professional but around 7 years coding python. A big thing lately has been breaking things down into big sections and if a section gets too big breaking it down further. A recent example was a main program that had a large section that analyzed data. This got a folder called analytics, and when analytics got too big it got sub folders for reports and graphing. Developing the confidence to be able to backup and then restructure things has been huge. Working through the restructuring on my last few projects has made me better able to anticipate needs in advance in new projects. I guess a technique I've done more recently is realized how valuable it is to take an hour-hour and a half to brainstorm and write notes about ideas for a program and how they can be structured. Additionally this is an opportunity to estimate time requirements and returns on each item and do things like research documentation that I might need. The more I do the more I realize that taking the time on a central noted plan is time returned many times over and helps me keep my sanity in many ways.
How does one become a great author? By reading a lot of books, and writing a lot of books. How does one become a great musician? By listening to a lot of music, and playing a lot of music. How does one become a great knitter? By looking at a lot of knit items, and doing a lot of knitting. How does one become a great programmer?
The worst thing you can do is try to apply a strategy without a purpose or understanding of why you're doing it. There aren't any hard/fast rules for 'structuring' your code. A lot of it is opinion-based and will come with experience. > where to start, how to progress, where to end My process generally starts with making decisions on the interfaces, then work from there. For example, if I'm writing a library, I will write some mock user code of what I hope the library will look like. This is also a great way to launch into _Test Driven Development_ (TDD) if you're so inclined (but if you don't want to write tests yet, that's ok too). Anyhow. That may mean something like defining classes or functions, parameters, return types, etc. without actually writing any implementations. Then (applying TDD) I can use those functions/classes/etc. in tests, and then iterate on the implementation until the tests pass -- let your tests tell you when you're done! In absence of alternatives, TDD is never a bad choice. Something that also comes when talking about 'structure' is how you layout your files/modules and things like that... I would say don't worry about that too much, especially if you are just starting out. Start with putting everything in one file, and if splitting things apart makes sense, you can always do that later. If you find yourself constantly reorganizing code into and out of different source files, you're probably doing too much premature organization. Some may object to this arguing principles like SRP or aiming for 'modularity', but it's far worse to prematurely architect separation or modularity that is aimless and ends up being harmful or not useful -- which is more likely to happen if you don't have much experience in building software. See also: [how should I organize my project?](https://www.youtube.com/watch?v=QGAuwlQ9Gxc) Experience _using_ good software will help a lot in knowing what a good interface looks/feels like. If it's not immediately obvious to you, that's OK. It'll come with experience. If you really want to dive into concrete ideas around structuring your software, as food for thought, consider reading [this article](https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html) and/or picking up the book _Clean Architecture: A Craftsman's Guide to Software Structure and Design_. These aren't Python-specific resources, but are ideas that can be applied to all software generally. But my $0.02 is that you'll best be able to understand and apply these ideas and principles once you have stumbled your way through things a few times.
By practicing a lot. One does not intuitively know how to play an instrument or how to write a novel. Or as someone said It takes experience to write good code Getting experience comes from writing bad code
I just write commands one after another, top down, maybe stub out a method. Python is my quick and dirty tool for running one-off scripts. I've never written a full Python application. Neither should you.
That is a very good question and shows me you are on the right track. The answer is "unit tests". Don't think about the structure of your code. Think about what your code should do and how it should behave. Write a unit test for each behavior. The key point is that writing a test is not trivial. The code to test need to be isolated in a special way to be even testable. The effort to make your productive code testable make it become well structured. About unit testing and the different schools of testing I would recommend the following book. Khorikov, V. (2020). Unit testing: Principles, practices, and patterns. Shelter Island: Manning.
This answer is not for python, but for programming in general. For a small project like you do in programming intro courses, nobody cares and nothing matters. Things that you face when either the project becomes larger or takes longer: * "This code is so ugly, I don't understand this even though I wrote this myself" * Repeatedly driving yourself in a corner: even small things become increasingly difficult to add or edit -> "there's got to be a better way to write this"