You touch upon some interesting pain points. I really like Julia and working with it is a pleasure.

Except the Module system, which feels unnecessarily arcane. I'm happy to be educated on why, but it seems to successfully combine the awkwardness of C-style #include with the mess of a free-form module system. The end result is a Frankenstein monster where technically everything is possible, everything could be included anywhere, there are no boundaries or even conventions. It makes for a frustrating experience for a newbie.

Say you have a package, and inside is a file called `xyz.jl`. You open the file and it defines a module called Xyz. But this tells you absolutely nothing about where in the package Xyz will appear. It could be included somewhere deep in the hierarchy, or it could be a submodule. It could be included multiple times in multiple places! That's bad design for sure, but the language places no boundaries on you. You open another file `abc.jl`, and see no modules at all, just a bunch of functions, which in turn call other functions that are defined God knows where. A julia file does not have to contain any information about where the symbols it's using come from, since it will be just pasted in verbatim to some location somewhere.

The whole module system feels like one big spaghetti of spooky action at a distance.

It's a shame too, because the rest of the language is very neat. Once one gets over the hurdle of the modules, it is possible to establish conventions to bring some sanity in there, but it's a hurdle that many people will probably not want to deal with.

How would you like modules to work?

It seems great to me that paths & source files are mostly irrelevant, you're free to re-organise without changing anything. And that `using Xyz` is always talking to the package manager. You can make sub-modules and say `using .Xyz`, but there's very little need to do so, and few packages do.

You can shoot yourself in the foot by including source twice, as you can by generating it with a macro, or simply copy-pasting wrong.

I mean, I'd like them to work like Python, Ruby and TypeScript, but you're right to say I can't describe why I want this.

Is there some guide I could read about structuring a large Julia project? It was pretty easy to intuit with Python, wherein I would put related files in a folder. But with Julia, everything is everywhere and I'm baffled.

Just reading what everyone does may work. Here's a tiny package:

https://github.com/ChrisRackauckas/EllipsisNotation.jl

I think that `/src/Name.jl` must have the main module, and `/test/runtests,jl` tests. And the package manager cares about `Project.toml`. But beyond this there are no real rules enforced, although there really seems to be one way to do things.

Here's a much bigger project, organised the same way. `include(file.jl)` literally copies in the text, and it's somewhat conventional to collect all imports & exports in the main file:

https://github.com/JuliaNLSolvers/Optim.jl/blob/master/src/O...

Still no sub-modules. No files included in mysterious locations. Methods being defined for functions from elsewhere are all qualified, like `Base.show(io::IO, t::OptimizationState) = ...`

> But with Julia, everything is everywhere and I'm baffled.

This is exactly it. Julia allows you to import and include anything, anywhere. You open a file and it doesn't say anything about where the dependencies are coming from and where this particular piece of code will go. Both of those are defined at the place where this file is included, which itself could be anywhere. It could be a different directory, different file, tucked away in a module. It could be in a dozen other files, or no files at all, and you can't tell from looking at just the source of the file.

In practice most packages:

1. Never use submodules (they don't add much) 2. Only use `include` within the main `src/MyPackage.jl` file

Languages like Python, Rust, C# or even Java have module systems that I find are more restrictive, but much easier to follow. You always have the pertinent information at hand. Each file containing code clearly tells you two crucial pieces of information:

1. Where the code fits in the greater picture

2. Where the dependencies of the code in a file come from

Python, whose module story is actually pretty poor, is still easier to follow than Julia, because it just matches the file/directory structure. You can reason about the hierarchy of a python library by just navigating the directories. In a normal python project, each file is one module and it's dependencies are clearly specified as imports.

Rust relies on the file system as well, and much better defined rules than Python. I find this great, because the file system is already hierarchical and we are used to the way it works. When I open a file in a Rust project, I know immediately where it fits in the hierarchy - because it is implied from the file system. Rust gives you a bit more flexibility in that you can define submodules in each file.

C# & Java qualify the namespaces fully in each file. While the file structure is not as clear anymore at the file system level, a single file contains all the information necessary to determine where the code fits in and where it's dependencies come from.

Now let's take Julia:

A single module will be often split across multiple files. Since they share a single namespace, the imports happen at the top level where the module is defined and includes all it's source files. When you open a source file, you have zero information where all the functions and data types are coming from (or where they are going for that matter).

I see the following pattern systematically emerge in Julia code:

- A function is defined in file A

- File A is `included` in file B, where it forms part of module X

- It is then imported into module Y in file C, but it is not actually used there

- As it is finally used in file D, which is `included` in module Y in file C itself

The problem is that there is no link from file A to file B, or module X for that matter. File A could be part of a dozen modules, or zero. Neither is there a link between the usage of the function in file D and where it is coming from. You actually have to find all the places where file D is included, and then check what flavor of the function does each location import. The relationships are established at some indeterminate level in the hierarchy.

Again, don't get me wrong, this is just a wart on an otherwise very pleasant language. I wouldn't be complaining if I weren't using it.

That was helpful, especially realizing that part of the problem (at least as you see it) is that the "linking" is unidirectional.

Function definitions/calls are also unidirectionally linked. You can see at the call site which function is called, but you can't see at the function definition the references. But unlike a function, which might be called from many places, it really should be the case that a file is `include`d exactly once.

Author of post here: there were a major gripe for me starting out too. It took me a fair while to conclude that they allowed to useful things in the bigger picture. and it certainly is not a pure win.

I do miss static typing.

I would question the claim it doesn't scale to production. I know people who have build hugely complex production systems in perl that are still running today 20 years later.

Further, I myself work on what we believe to be the largest closed source julia code base, in terms of number of contributors, number of packages and total size. (Its also pretty large in general, though i have yet to work out how it stacks up against DiffEq-verse). And I have seen thing go from research prototype into running in production. It works.

I am not going to deny though there are advantages to other languages. There are many trade-offs in the world

> I know people who have build hugely complex production systems in perl that are still running today 20 years later.

Sure, but there aren't many programmers who would want to maintain such a system.

Perl's just another language I've known. Maintaining a 20yo codebase in a language that people find distasteful sounds like comfortable job security, to me.

Most programmers want to build new things, not work on maintenance projects, regardless of tech stack.

That said, many people do enjoy that sort of work and finding them is not unusually difficult unless the project is so big that you need dozens of bodies.

People write large-scale systems in dynamically-typed languages all the time. Multiple dispatch and macros make clean scaling easier than it would be in most other dynamic languages. Its competitors in numerical performance are C/C++ and Fortran, which are both minefields (C much more so). Julia is definitely safer in practice than these kind of languages with weak, unsafe type systems.

I'm not saying static types don't have benefits as well, but it would also be very against the design goals as a Matlab/R competitor.

Inheritance would also directly clash and overlap with multiple dispatch, which is strictly more powerful.

> I'm not saying static types don't have benefits as well,

It's funny; formerly I was a die-hard fan of static typing, but, lately, my opinions have become more nuanced. Something more along the lines of, "Yeah, I'd never want to just remove the type annotations from my Java code and then try to maintain the result, but some dynamic languages allow me to have a fraction as much code to maintain in the first place."

I'm also beginning to wonder if my feelings about dynamic languages have been unduly influenced by some particularly popular, and also particularly undisciplined, dynamic languages. JavaScript and PHP, for example.

> It's funny; formerly I was a die-hard fan of static typing, but, lately, my opinions have become more nuanced. Something more along the lines of, "Yeah, I'd never want to just remove the type annotations from my Java code

That might be one of the issues. Even when it comes to static typing java is high investment low rewards.

Things are admittedly less bad than they were 15 years ago, but it remains that java has very high overhead both syntactically and at runtime and yet is pretty limited in its static expressivity. So when you do away with java for a dynamically typed langage most of what you lost from missing static types you gain back from having so much less LOCs and ceremony and architectural astronautics to deal with.

In theory you could probably write terser java but that’s not what the community does or what the ecosystem encourage, so if you do you’re on your own and then why keep using java? There’s plenty of better langages with no community and no ecosystem out there. And they have way, way better type systems.

> Even when it comes to static typing java is high investment low rewards.

My sense is that it's because Java isn't really all that static a language. In terms of the syntax, sure. But it also relies very heavily on run-time reflection, run-time type casting, and run-time type checking. Since the introduction of generics, there are even some situations where the types are known and declared statically in the source code, but the compiler is unable to do static type verification, and so the type checking only happens dynamically at run time.

Meaning that you kind of get the worst of both worlds: High-ceremony programming, but not a whole lot of help from the compiler in return for it.

Anyway, yeah, that does mean that the debate does get more hair-splitty if we're talking Haskell vs. Racket. But, realistically, that's not usually the languages people have in mind when they're talking static vs dynamic - it's much more likely to be Java or Python. Both of which, like most languages, fall short of being the ideal examples of their respective type checking approaches.

> My sense is that it's because Java isn't really all that static a language.

All the stuff that happens at runtime explains the "low rewards" part, but it doesn't explain the "high investment", Java is also an extremely verbose language, doing anything is verbose and the one thing you'd want to be doing (create and use new types) is one of the most verbose parts of the language.

Records will eventually make things less bad, but a "newtype pattern" in Java takes a dozen lines or five, that's horrendous for something which should take maybe two lines. And that's before we even consider the insanity of the "one (public) class per file" mandate.

> But, realistically, that's not usually the languages people have in mind when they're talking static vs dynamic - it's much more likely to be Java or Python. Both of which, like most languages, fall short of being the ideal examples of their respective type checking approaches.

There are lots of criticisms you can leverage at Python, but it's nowhere near as bad a representative of dynamically typed language as Java is statically typed ones.

Java's code verbose-ness is not entirely due to static typing though. IMHO, it is ecosystem, people, SO-copy-culture

> multiple dispatch, which is strictly more powerful.

In a language that supports classes I can have class B inherit from class A and automatically provide all of class A's functionality without adding a single extra line of code. I can extend class B's functionality by adding only code that is specific to it.

I don't see how to do that with multiple dispatch, at least the way it's implemented in Julia.

>In a language that supports classes I can have class B inherit from class A and automatically provide all of class A's functionality without adding a single extra line of code.

And people will abuse this to create horrible brittle inheritance hierachies. There's a reason modern languages like Go and Rust deliberately don't support implementation inheritance; to many people it's an antifeature.

The thing about software is that the "one true way" changes every 2 or 3 years.

A few decades ago inheritance was considered a key software design principle, then it was abused by many (especially Java programmers, I think), just like any other powerful feature, now it's considered evil. If multiple dispatch becomes as popular as classes/inheritance was, I suspect it will go through the same love/hate cycle.

All I know is that I have used implementation inheritance to good effect on multiple occasions and found it to be a very valuable feature.

Inheritance is not really all that powerful, assuming that one follows SOLID and especially Liskov substitution. There's very few places where it's actually applicable. Interfaces are much more widely applicable -- defining contracts instead of behaviour makes it much easier to follow SOLID.

I mostly only use inheritance for patching the behaviour of some code I don't own by overriding some specific method. Because that's the only mechanism the language provides to perform such a modification. This is basically the use-case that Julia's multiple dispatch addresses.

It's important to always remember the context that SOLID is a collection of one man's opinions.

When we mix classes to create a new class, the ingredients in the mixture retain their adherence to their respective contracts, and we didn't have to re-implement them.

> I mostly only use inheritance for patching the behaviour of some code.

That's good for you, but you have to realize that classes specifically designed as inheritance bases are also a thing and the experience of using those kinds of classes with inheritance is not the same as deriving from any random class whose behavior we don't like in some aspect.

The conventions by which a class supports inheritance are also a form of contract! The contract says that if you derive from me, you can expect certain behaviors, and also have certain responsibilities regarding what to implement and how, and what not to depend on or not to do and such.

If a class is not designed for inheritance, then there is no contract, beyond some superficial expectations that come from the OOP system, some of which can fall victim to hostilities in the way the class is implemented, or maintained in the future.

It's interesting to think about what multiple dispatch abuse would be.

As far as I know, implementation inheritance can be misapplied when you try too hard to use the language's type system to capture some domain-specific model. e.g. you're dealing with multiple kinds of entities, and they all have a `name`, so you decide that all of the classes should inherit from `class Named`, because they have that in common with each other.

Well, there's lots of different taxonomies possible for your entities, probably. And single implementation inheritance lets you express just one taxonomy. Interfaces and multiple inheritance aim to allow multiple co-existing taxonomies, but I can't really explain the ways in which its misapplied.

I agree that there are some clear situations for implementation inheritance, so it seems like a useful pattern to be able to support. I think the discussion is whether it should be a key design principle, and built into the language design, or if the language design should be more general, so that implementation inheritance can be built on top of that.

In TXR Lisp, I doubled down on OOP and extended the object system to support multiple inheritance, just this December.

Real inheritance of code and data allows for mixin programming, which is very useful.

If you don't give OOP programmers the tools for mixing programming, they will find some other way to do it, such as some unattractive design pattern that fills their code with boilerplate, secondary objects, and unnecessary additional run-time switching and whatnot, all of which can be as brittle as any inheritance hierarchy, if not more.

Single-dispatch OOP dispatches (implicitly) only on the first argument, self. That is how Class B can provide the functionality of A. Multiple dispatch can dispatch on all arguments. Thus, OOP single dispatch is a special case of multiple dispatch.

In your example, functionality in class B can be written simply as ordinary generic functions, which are inferred to their most general compatible type without annotations. Because of this, all functionality for class A will work for class B, as long as B is a superset of A. Not a single line of code, and no brittle inheritance hierarchies.

Multiple dispatch is one solution to the expression problem, which both OOP and functional programming suffer from in dual ways. Other ones include Haskell's typeclasses and Ruby's mixins.

Yes, but in Julia the problem is that inheritance isn't allowed from concrete types. So this example in Julia would mean that A is an abstract type whose objects can contain no data. So you have to figure out a way to implement functionality on A without having any data to work with. There are several ways to work around this but they all involve considerably more code complexity than would be required in a language like C++ that supports classes.

It’s maybe less of a problem than you think. If you simply want to call a method on two different types you can often just do it. Type annotations generally don’t help with performance, and any type checking is only going to happen at run time anyway.

Inheritance does help with documenting what functions are compatible with what types. I think that could be better in Julia, and things like abstract type hierarchies and traits help a bit. Concrete inheritance could be nice, but that seems to also enable some pretty bad OOP practices.

This can be done better with traits in Julia, which are more powerful, safer and faster.

See here: https://white.ucc.asn.au/2018/10/03/Dispatch,-Traits-and-Met...

Perhaps. The feeling I get with Julia is that the devs are sort of making it up as they go along. I don't mean just making up the language (which of course they're doing), but making up whole new approaches to programming that aren't necessarily well-understood and tested in the real world and certainly aren't very familiar to most programmers.

Maybe in the end they will succeed and will invent a generally superior approach to software development. But for the moment it all feels very experimental and so not a language I'd want to commit to at present if I were starting a major, non-solo project.

A lot of the ideas come from Common Lisp.

Generally via wrapping (Composition).

``` struct Foo <: AbstractFoo ... end

struct SpecialFoo{T<:AbstractFoo} <: AbstractFoo backing::T end ```

and then delegating calls to the wrapped object

Yes, but delegation requires a lot of extra code to get what you have for free in a class based language. And yes I know you can probably write macros to handle much of that but having to use meta-programming to get what other languages give you for free doesn't seem ideal to me.

If using meta-programming to get "basic" features rubs you the wrong way, the language might just not be for you. In general, the Julia philosophy (as well as the lisp philosophy from which Julia descends) is that things which can be done efficiently using macros instead of being built-in should be done with macros. Language features are only for things that cannot be accomplished via metaprogramming. As far as actual implementations of method-forwarding macros, there's an implementation in Lazy.jl[1], and TypedDelegation.jl[2].

[1] https://github.com/MikeInnes/Lazy.jl/blob/0de1643f37555396d6...

[2] https://github.com/JeffreySarnoff/TypedDelegation.jl

I have concluded that it is basically impossible to write a general purpose delegation macro.

I have seen many and alway immediately run into cases they can't handle.

A top level `for m in (:foo, :bar); @eval $m(X::MyType) = $m(X.backing)`

Is general and has no performance overhead

To be honest, I've never found the need for delegating a huge number of methods in my own work, but then I've never wanted to add a feature or field to something as complex and featureful as e.g. a DataFrame. What issues have you run into when using method-forwarding/delegation macros?

The nice thing about the Julia ecosystem is that people tend to be pretty willing to step back and define their methods in terms of an interface (Tables.jl in this case) which then allows code reuse without brittle delegation. Having to put so much effort into changing the structure of upstream doesn't seem ideal from a composability perspective though.

You are not wrong. Also even the best metaprogramming solution will not help you you when you method to add to your backing type

I don't understand type systems very well. When you say "check correctness" do you mean something beyond static linting with type-hints, like in Python? Or do you mean something deeper, like in functional languages like Elm and F#?

Also, is there always a trade-off between types and flexible meta-programming? Like, OCaml has meta-programming capabilities, but they make type-checking way harder, according to my PhD friend who's written extensively in Scheme and OCaml.

Yep this has been my conclusion as well, I really wanted to like Julia and there are parts of it I do, but I think it misses the mark in some big ways.