Option dfs.client.use.datanode.hostname is ignored

[faladu]

The library ignores the setting dfs.client.use.datanode.hostname
This setting should make the library use names to connect to datanodes instead of the ipaddress.

In this library getIpAddr() is always used instead of using getHostName() if this setting is true.

If you have a cluster where the datanodes use internal ip-adresses in the configuration files but also have external ip-addresses for client access the getIpAddr always returns the internal ones. In this case you'd want to use the option so you get the names instead which the workstation the client runs on then resolves to the external ip adresses.

I made some adjustments on my download of this repository to enable this option and it seems to work well.
I wanted to ask if there is some way for me to commit them for inspection so they can make it back into this repository over time.

Kind regards
Max

[martindurant]

Note that this repo has had no commits in a year and a half. The build and test process for the various possible hadoop setups proved difficult, since we are python developers, not c++. Generally for Dask and at Anaconda in general, we are suggesting people use arrow.hdfs instead.

You are however totally welcome to submit a PR for the good of the community. I can merge it, without even necessarily understanding it. I don't know if it would build successfully - and you should note that the conda recipe currently pulls from my personal fork, not this version, but this could be changed https://github.com/conda-forge/libhdfs3-feedstock/blob/master/recipe/meta.yaml#L11

[faladu]

Hi there,

thanks for the info.
My problem is that i need to write to hdfs using c and even though the api is a bit outdated it still seems to vastly outperform the jni wrapper libhdfs so I continued using it.

This github was what was linked most often when searching for libhdfs3 so I used that one as my base version when I started working with it.

I'll look up how I do a pull request to commit my changes, so far I only ever downloaded from github and used it never commited back.

What the change does is that it enables the option "dfs.client.use.datanode.hostname"
This option is set to false as default and in that case nothing changes.
If you set this option to true after the changes it would use the hostname to connect to the datanode instead of still using the ip address like it does now.

So it should only be an enhancement for whoever wants to use that option and do nothing for anyone who doesn't.

[pulsarsitra]

Hi @faladu

I am facing the same problem in one of my deployment. Could you please provide the code diff to me, it will save lot of my time.

pulsarsitra@gmail.com

Thanks,
Sitrarasu J